[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:34 dist-test-slave-j663 krb5kdc[11765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:34 dist-test-slave-j663 krb5kdc[11765](info): set up 2 sockets
Jun 20 07:16:34 dist-test-slave-j663 krb5kdc[11765](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 07:16:35.089011 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:16:35 dist-test-slave-j663 krb5kdc[11765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:16:35.204136 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:46657
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:46657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:35.309940 11778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:35.310173 11778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:35.310210 11778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:35.314136 11778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:35.314188 11778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:35.314206 11778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:35.314224 11778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:35.314239 11778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:35.319222 11778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:46657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:46657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:35.320107 11778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:35.321018 11778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:16:35.329416 11778 server_base.cc:1034] running on GCE node
W20260620 07:16:35.330229 11783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:35.330302 11784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:35.330447 11786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:35.330940 11778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:35.331938 11778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:35.333148 11778 hybrid_clock.cc:648] HybridClock initialized: now 1781939795333133 us; error 28 us; skew 500 ppm
Jun 20 07:16:35 dist-test-slave-j663 krb5kdc[11765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939795, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:35.335857 11778 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:16:35.336733 11778 webserver.cc:460] Webserver started at http://127.11.123.190:37219/ using document root <none> and password file <none>
I20260620 07:16:35.337324 11778 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:35.337414 11778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:35.337638 11778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:35.339591 11778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "6aee9d83644b4e65a9b9f11619d44244"
format_stamp: "Formatted at 2026-06-20 07:16:35 on dist-test-slave-j663"
server_key: "681161ca1b42ec1dc5fa0ed8ff82fda6"
server_key_iv: "86183740895a5260e39aad21a2eed305"
server_key_version: "encryptionkey@0"
I20260620 07:16:35.340160 11778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "6aee9d83644b4e65a9b9f11619d44244"
format_stamp: "Formatted at 2026-06-20 07:16:35 on dist-test-slave-j663"
server_key: "681161ca1b42ec1dc5fa0ed8ff82fda6"
server_key_iv: "86183740895a5260e39aad21a2eed305"
server_key_version: "encryptionkey@0"
I20260620 07:16:35.343690 11778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:16:35.345954 11793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:35.347069 11778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:16:35.347204 11778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "6aee9d83644b4e65a9b9f11619d44244"
format_stamp: "Formatted at 2026-06-20 07:16:35 on dist-test-slave-j663"
server_key: "681161ca1b42ec1dc5fa0ed8ff82fda6"
server_key_iv: "86183740895a5260e39aad21a2eed305"
server_key_version: "encryptionkey@0"
I20260620 07:16:35.347309 11778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:35.362346 11778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:35.365608 11778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:35.365808 11778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:35.373060 11778 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:46657
I20260620 07:16:35.373093 11845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:46657 every 8 connection(s)
I20260620 07:16:35.374008 11778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:35.376624 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:35.379705 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 11778
I20260620 07:16:35.379873 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:16:35.381556 11758 external_mini_cluster.cc:1428] Setting key 423b4be03168c637efd024f2d5a8d78c
I20260620 07:16:35.382822 11846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244: Bootstrap starting.
I20260620 07:16:35.385663 11846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:35.386478 11846 log.cc:824] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:35.388655 11846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244: No bootstrap required, opened a new log
I20260620 07:16:35.393477 11846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aee9d83644b4e65a9b9f11619d44244" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46657 } }
I20260620 07:16:35.393750 11846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:35.393800 11846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aee9d83644b4e65a9b9f11619d44244, State: Initialized, Role: FOLLOWER
I20260620 07:16:35.394261 11846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aee9d83644b4e65a9b9f11619d44244" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46657 } }
I20260620 07:16:35.394418 11846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:35.394482 11846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:35.394604 11846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:35.395447 11846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aee9d83644b4e65a9b9f11619d44244" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46657 } }
I20260620 07:16:35.395771 11846 leader_election.cc:302] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aee9d83644b4e65a9b9f11619d44244; no voters:
I20260620 07:16:35.396131 11846 leader_election.cc:288] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:35.396247 11851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:35.396534 11851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [term 1 LEADER]: Becoming Leader. State: Replica: 6aee9d83644b4e65a9b9f11619d44244, State: Running, Role: LEADER
I20260620 07:16:35.396836 11851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aee9d83644b4e65a9b9f11619d44244" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46657 } }
I20260620 07:16:35.397384 11846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:35.398420 11852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aee9d83644b4e65a9b9f11619d44244" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aee9d83644b4e65a9b9f11619d44244" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46657 } } }
I20260620 07:16:35.398561 11852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:35.398849 11853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aee9d83644b4e65a9b9f11619d44244. Latest consensus state: current_term: 1 leader_uuid: "6aee9d83644b4e65a9b9f11619d44244" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aee9d83644b4e65a9b9f11619d44244" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46657 } } }
I20260620 07:16:35.398936 11853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:35.399209 11860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 07:16:35 dist-test-slave-j663 krb5kdc[11765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:35.402637 11860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:35.408605 11860 catalog_manager.cc:1347] Generated new cluster ID: 17fd15fc629740c8a30a8920448d2ee2
I20260620 07:16:35.408669 11860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:35.408962 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:35.385152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.11.123.190:46657)
0620 07:16:35.385665 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:35.385686 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:35.386745 (+ 1059us) server_negotiation.cc:407] Connection header received
0620 07:16:35.391072 (+ 4327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:35.391087 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:35.391374 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:35.391631 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:35.394620 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:35.395716 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:35.397374 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:35.398087 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:35.398271 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:35.403427 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:35.403434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:35.403459 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:35.403491 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:35.405607 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:35.406234 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:35.406239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:35.406243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:35.406345 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:35.406784 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:35.406787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:35.406788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:35.407139 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:35.407364 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:35.407868 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 07:16:35.408114 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":125,"threads_started":1}
I20260620 07:16:35.422134 11860 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:35.422971 11860 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:35.437040 11860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aee9d83644b4e65a9b9f11619d44244: Generated new TSK 0
I20260620 07:16:35.437811 11860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:16:35.468358 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46657
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:35.576396 11872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:35.576634 11872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:35.576678 11872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:35.580557 11872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:35.580607 11872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:35.580664 11872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:16:35.585639 11872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46657
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:35.586508 11872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:35.587376 11872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:35.596313 11878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:35.596405 11880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:35.596414 11877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:35.596714 11872 server_base.cc:1034] running on GCE node
I20260620 07:16:35.597134 11872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:35.597856 11872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:35.599037 11872 hybrid_clock.cc:648] HybridClock initialized: now 1781939795599020 us; error 29 us; skew 500 ppm
Jun 20 07:16:35 dist-test-slave-j663 krb5kdc[11765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939795, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:35.601905 11872 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:16:35.602636 11872 webserver.cc:460] Webserver started at http://127.11.123.129:38347/ using document root <none> and password file <none>
I20260620 07:16:35.603099 11872 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:35.603155 11872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:35.603304 11872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:35.605201 11872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "2062a913f9d44b8198d096ad60a18195"
format_stamp: "Formatted at 2026-06-20 07:16:35 on dist-test-slave-j663"
server_key: "bf6ae73493a1e870b8cd0ae69d85092e"
server_key_iv: "e2afe6199e2e5032c86b44b8fc4ef056"
server_key_version: "encryptionkey@0"
I20260620 07:16:35.605690 11872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "2062a913f9d44b8198d096ad60a18195"
format_stamp: "Formatted at 2026-06-20 07:16:35 on dist-test-slave-j663"
server_key: "bf6ae73493a1e870b8cd0ae69d85092e"
server_key_iv: "e2afe6199e2e5032c86b44b8fc4ef056"
server_key_version: "encryptionkey@0"
I20260620 07:16:35.609014 11872 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:16:35.611305 11887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:35.612486 11872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:16:35.612624 11872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "2062a913f9d44b8198d096ad60a18195"
format_stamp: "Formatted at 2026-06-20 07:16:35 on dist-test-slave-j663"
server_key: "bf6ae73493a1e870b8cd0ae69d85092e"
server_key_iv: "e2afe6199e2e5032c86b44b8fc4ef056"
server_key_version: "encryptionkey@0"
I20260620 07:16:35.612721 11872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:35.634346 11872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:35.637574 11872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:35.637784 11872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:35.638372 11872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:35.639175 11872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:35.639242 11872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:35.639313 11872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:35.639355 11872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:35.648054 11872 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:32919
I20260620 07:16:35.648085 12000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:32919 every 8 connection(s)
I20260620 07:16:35.648898 11872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:16:35.654117 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 11872
I20260620 07:16:35.654229 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:16:35.654459 11758 external_mini_cluster.cc:1428] Setting key 9540cd1eb98bc25a92e720ccb7af2304
Jun 20 07:16:35 dist-test-slave-j663 krb5kdc[11765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939795, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:35.663089 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:35.650729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:54941 (local address 127.11.123.190:46657)
0620 07:16:35.650858 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:35.650862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:35.651713 (+ 851us) server_negotiation.cc:407] Connection header received
0620 07:16:35.652996 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:35.652999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:35.653044 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:35.653200 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:35.654271 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:35.654931 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:35.656380 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:35.656644 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:35.656727 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:35.659328 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:35.659335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:35.659341 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:35.659373 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:35.661113 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:35.661577 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:35.661583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:35.661587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:35.661680 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:35.661998 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:35.662002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:35.662006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:35.662231 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:35.662344 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:35.662773 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 07:16:35.662905 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:16:35.663757 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:35.651154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46657 (local address 127.11.123.129:54941)
0620 07:16:35.651570 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:35.651609 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:35.652804 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:35.653331 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:35.653337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:35.653670 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:35.654123 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:35.654136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:35.655065 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:35.655069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:35.656252 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:35.656266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:35.656792 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:35.656798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:35.657020 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:35.657752 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:35.657767 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:35.659162 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:35.661225 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:35.661229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:35.661239 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:35.661474 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:35.661776 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:35.661778 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:35.661779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:35.661899 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:35.662334 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:35.662339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:35.662542 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 07:16:35.662688 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:16:35.662911 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:16:35.664917 12001 heartbeater.cc:347] Connected to a master server at 127.11.123.190:46657
I20260620 07:16:35.665195 12001 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:35.665618 12001 heartbeater.cc:511] Master 127.11.123.190:46657 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
I20260620 07:16:35.667227 11810 ts_manager.cc:194] Registered new tserver with Master: 2062a913f9d44b8198d096ad60a18195 (127.11.123.129:32919)
I20260620 07:16:35.668560 11810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:54941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:16:35.681877 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46657
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:35.787734 12006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:35.788029 12006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:35.788108 12006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:35.792223 12006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:35.792315 12006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:35.792412 12006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:16:35.797499 12006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46657
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:35.798575 12006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:35.799556 12006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:16:35.808651 12006 server_base.cc:1034] running on GCE node
W20260620 07:16:35.808558 12012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:35.808508 12014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:35.808530 12011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:35.809314 12006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:35.809921 12006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:35.811107 12006 hybrid_clock.cc:648] HybridClock initialized: now 1781939795811072 us; error 46 us; skew 500 ppm
Jun 20 07:16:35 dist-test-slave-j663 krb5kdc[11765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939795, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:35.813784 12006 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:16:35.814531 12006 webserver.cc:460] Webserver started at http://127.11.123.130:33111/ using document root <none> and password file <none>
I20260620 07:16:35.815037 12006 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:35.815091 12006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:35.815245 12006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:35.817375 12006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "f679aa44c88242e781b55ed9d0876dba"
format_stamp: "Formatted at 2026-06-20 07:16:35 on dist-test-slave-j663"
server_key: "b937da215a0858bd6d9165f6ae369715"
server_key_iv: "6a28eb5842f2428eae16cf4884716cd3"
server_key_version: "encryptionkey@0"
I20260620 07:16:35.817880 12006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "f679aa44c88242e781b55ed9d0876dba"
format_stamp: "Formatted at 2026-06-20 07:16:35 on dist-test-slave-j663"
server_key: "b937da215a0858bd6d9165f6ae369715"
server_key_iv: "6a28eb5842f2428eae16cf4884716cd3"
server_key_version: "encryptionkey@0"
I20260620 07:16:35.821384 12006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:16:35.823642 12021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:35.824903 12006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:35.825011 12006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "f679aa44c88242e781b55ed9d0876dba"
format_stamp: "Formatted at 2026-06-20 07:16:35 on dist-test-slave-j663"
server_key: "b937da215a0858bd6d9165f6ae369715"
server_key_iv: "6a28eb5842f2428eae16cf4884716cd3"
server_key_version: "encryptionkey@0"
I20260620 07:16:35.825157 12006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:35.842918 12006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:35.846235 12006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:35.846449 12006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:35.847067 12006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:35.847927 12006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:35.847976 12006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:35.848048 12006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:35.848088 12006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:35.857223 12006 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:38369
I20260620 07:16:35.857245 12134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:38369 every 8 connection(s)
I20260620 07:16:35.858129 12006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
Jun 20 07:16:35 dist-test-slave-j663 krb5kdc[11765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939795, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:35.867408 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 12006
I20260620 07:16:35.867528 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:16:35.867834 11758 external_mini_cluster.cc:1428] Setting key 931df00b7022729747bb4fdc841cbd3f
I20260620 07:16:35.871991 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:35.859937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:37105 (local address 127.11.123.190:46657)
0620 07:16:35.860073 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:35.860076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:35.860948 (+ 872us) server_negotiation.cc:407] Connection header received
0620 07:16:35.862288 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:35.862292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:35.862349 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:35.862452 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:35.863560 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:35.864086 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:35.865224 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:35.865447 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:35.865559 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:35.868048 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:35.868052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:35.868055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:35.868082 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:35.869771 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:35.870275 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:35.870282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:35.870287 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:35.870376 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:35.870799 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:35.870805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:35.870809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:35.871051 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:35.871165 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:35.871791 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 07:16:35.871881 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:16:35.872974 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:35.860351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46657 (local address 127.11.123.130:37105)
0620 07:16:35.860810 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:35.860851 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:35.862074 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:35.862618 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:35.862624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:35.862961 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:35.863414 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:35.863426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:35.864208 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:35.864213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:35.865047 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:35.865056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:35.865594 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:35.865598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:35.865769 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:35.866305 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:35.866320 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:35.867845 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:35.869908 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:35.869913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:35.869924 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:35.870171 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:35.870519 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:35.870521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:35.870523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:35.870693 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:35.871191 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:35.871200 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:35.871524 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 07:16:35.871744 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 07:16:35.872119 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":148,"threads_started":1}
I20260620 07:16:35.874200 12135 heartbeater.cc:347] Connected to a master server at 127.11.123.190:46657
I20260620 07:16:35.874444 12135 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:35.874920 12135 heartbeater.cc:511] Master 127.11.123.190:46657 requested a full tablet report, sending...
I20260620 07:16:35.875923 11810 ts_manager.cc:194] Registered new tserver with Master: f679aa44c88242e781b55ed9d0876dba (127.11.123.130:38369)
I20260620 07:16:35.876480 11810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:37105
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:16:35.895838 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46657
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:36.003563 12140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:36.003816 12140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:36.003885 12140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:36.008070 12140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:36.008157 12140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:36.008239 12140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:16:36.013849 12140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46657
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:36.014807 12140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:36.015837 12140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:36.025117 12146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:36.025174 12148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:36.025243 12140 server_base.cc:1034] running on GCE node
W20260620 07:16:36.025435 12145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:36.025811 12140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:36.026430 12140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:36.027616 12140 hybrid_clock.cc:648] HybridClock initialized: now 1781939796027594 us; error 31 us; skew 500 ppm
Jun 20 07:16:36 dist-test-slave-j663 krb5kdc[11765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939796, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:36.035043 12140 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:16:36.035889 12140 webserver.cc:460] Webserver started at http://127.11.123.131:44063/ using document root <none> and password file <none>
I20260620 07:16:36.036424 12140 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:36.036512 12140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:36.036751 12140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:36.038713 12140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "5aaf51b7887e46659e8b91d278383732"
format_stamp: "Formatted at 2026-06-20 07:16:36 on dist-test-slave-j663"
server_key: "db36215c0ce597e87935477f2da88c5b"
server_key_iv: "dee3b33e7b18450c3156a315ae927ffc"
server_key_version: "encryptionkey@0"
I20260620 07:16:36.039258 12140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "5aaf51b7887e46659e8b91d278383732"
format_stamp: "Formatted at 2026-06-20 07:16:36 on dist-test-slave-j663"
server_key: "db36215c0ce597e87935477f2da88c5b"
server_key_iv: "dee3b33e7b18450c3156a315ae927ffc"
server_key_version: "encryptionkey@0"
I20260620 07:16:36.042830 12140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:16:36.045149 12155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:36.046244 12140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:16:36.046375 12140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "5aaf51b7887e46659e8b91d278383732"
format_stamp: "Formatted at 2026-06-20 07:16:36 on dist-test-slave-j663"
server_key: "db36215c0ce597e87935477f2da88c5b"
server_key_iv: "dee3b33e7b18450c3156a315ae927ffc"
server_key_version: "encryptionkey@0"
I20260620 07:16:36.046480 12140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:36.077111 12140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:36.079888 12140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:36.080085 12140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:36.080653 12140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:36.081485 12140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:36.081562 12140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:36.081634 12140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:36.081671 12140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:36.090689 12140 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:44651
I20260620 07:16:36.090761 12268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:44651 every 8 connection(s)
I20260620 07:16:36.091538 12140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:16:36.092150 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 12140
I20260620 07:16:36.092253 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:16:36.092471 11758 external_mini_cluster.cc:1428] Setting key f11c0b7626cfbdc2531f6d550782a671
Jun 20 07:16:36 dist-test-slave-j663 krb5kdc[11765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939796, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:36.104637 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:36.093287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:40841 (local address 127.11.123.190:46657)
0620 07:16:36.093426 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:36.093433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:36.094435 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 07:16:36.095813 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:36.095815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:36.095860 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:36.095926 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:36.097001 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:36.097595 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:36.098866 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:36.099079 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:36.099169 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:36.101732 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:36.101735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:36.101737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:36.101760 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:36.103020 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:36.103449 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:36.103452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:36.103453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:36.103522 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:36.103794 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:36.103796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:36.103797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:36.103950 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:36.104022 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:36.104423 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 07:16:36.104509 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:16:36.105576 12271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:36.093731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46657 (local address 127.11.123.131:40841)
0620 07:16:36.094278 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:36.094330 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:36.095633 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:36.096082 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:36.096088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:36.096449 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:36.096886 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:36.096898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:36.097719 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:36.097722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:36.098761 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:36.098770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:36.099229 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:36.099235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:36.099444 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:36.100080 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:36.100095 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:36.101588 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:36.103120 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:36.103125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:36.103135 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:36.103365 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:36.103597 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:36.103599 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:36.103600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:36.103718 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:36.104031 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:36.104036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:36.104236 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 07:16:36.104378 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 07:16:36.104633 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":157,"threads_started":1}
I20260620 07:16:36.106593 12269 heartbeater.cc:347] Connected to a master server at 127.11.123.190:46657
I20260620 07:16:36.106791 12269 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:36.107184 12269 heartbeater.cc:511] Master 127.11.123.190:46657 requested a full tablet report, sending...
I20260620 07:16:36.108059 11809 ts_manager.cc:194] Registered new tserver with Master: 5aaf51b7887e46659e8b91d278383732 (127.11.123.131:44651)
I20260620 07:16:36.108560 11809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:40841
I20260620 07:16:36.118252 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:16:36 dist-test-slave-j663 krb5kdc[11765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939796, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:16:36 dist-test-slave-j663 krb5kdc[11765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939796, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.123.129@KRBTEST.COM
I20260620 07:16:36.143237 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:36.134294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.11.123.129:32919)
0620 07:16:36.134667 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:36.134680 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:36.134701 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:16:36.134746 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:36.134751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:36.134923 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:36.135036 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:36.135680 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:36.136426 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:36.137329 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:36.137694 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:36.137775 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:36.140063 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:36.140072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:36.140083 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:36.140114 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:36.141709 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:36.142102 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:36.142107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:36.142108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:36.142192 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:36.142423 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:36.142425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:36.142427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:36.142635 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:36.142778 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:36.142967 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 07:16:36.143081 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":178,"threads_started":1}
W20260620 07:16:36.144151 11915 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48238
Jun 20 07:16:36 dist-test-slave-j663 krb5kdc[11765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:16:36 dist-test-slave-j663 krb5kdc[11765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.129@KRBTEST.COM
I20260620 07:16:36.168326 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:36.160543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.11.123.129:32919)
0620 07:16:36.160688 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:36.160692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:36.160817 (+ 125us) server_negotiation.cc:407] Connection header received
0620 07:16:36.160978 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:36.160983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:36.161032 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:36.161142 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:36.161814 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:36.162320 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:36.163127 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:36.163360 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:36.163475 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:36.165751 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:36.165754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:36.165756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:36.165775 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:36.166990 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:36.167344 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:36.167346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:36.167347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:36.167412 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:36.167683 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:36.167685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:36.167686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:36.167826 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:36.167918 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:36.168124 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 07:16:36.168212 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:16:36.169360 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 11872
I20260620 07:16:36.175024 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 12006
I20260620 07:16:36.182188 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 12140
I20260620 07:16:36.187649 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 11778
2026-06-20T07:16:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2156 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:36 dist-test-slave-j663 krb5kdc[12283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:36 dist-test-slave-j663 krb5kdc[12283](info): set up 2 sockets
Jun 20 07:16:36 dist-test-slave-j663 krb5kdc[12283](info): commencing operation
krb5kdc: starting...
W20260620 07:16:37.244647 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:16:37 dist-test-slave-j663 krb5kdc[12283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:16:37.359596 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:33607
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:33607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:37.471280 12296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:37.471504 12296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:37.471582 12296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:37.475636 12296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:37.475684 12296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:37.475704 12296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:37.475747 12296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:37.475777 12296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:37.480772 12296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:33607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:33607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:37.481767 12296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:37.482717 12296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:37.490906 12304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:37.490917 12302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:37.491043 12296 server_base.cc:1034] running on GCE node
W20260620 07:16:37.491250 12301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:37.491757 12296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:37.492796 12296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:37.493949 12296 hybrid_clock.cc:648] HybridClock initialized: now 1781939797493907 us; error 52 us; skew 500 ppm
Jun 20 07:16:37 dist-test-slave-j663 krb5kdc[12283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939797, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:37.496620 12296 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:16:37.497587 12296 webserver.cc:460] Webserver started at http://127.11.123.190:42841/ using document root <none> and password file <none>
I20260620 07:16:37.498138 12296 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:37.498227 12296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:37.498451 12296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:37.500437 12296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "530ab13801e048578945e66a2fbec9a7"
format_stamp: "Formatted at 2026-06-20 07:16:37 on dist-test-slave-j663"
server_key: "25deb63c75e05b85860818fa228f79a9"
server_key_iv: "d258baa359bd0901f473cbdb1406be62"
server_key_version: "encryptionkey@0"
I20260620 07:16:37.500994 12296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "530ab13801e048578945e66a2fbec9a7"
format_stamp: "Formatted at 2026-06-20 07:16:37 on dist-test-slave-j663"
server_key: "25deb63c75e05b85860818fa228f79a9"
server_key_iv: "d258baa359bd0901f473cbdb1406be62"
server_key_version: "encryptionkey@0"
I20260620 07:16:37.504632 12296 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:16:37.507190 12311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:37.508499 12296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:37.508623 12296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "530ab13801e048578945e66a2fbec9a7"
format_stamp: "Formatted at 2026-06-20 07:16:37 on dist-test-slave-j663"
server_key: "25deb63c75e05b85860818fa228f79a9"
server_key_iv: "d258baa359bd0901f473cbdb1406be62"
server_key_version: "encryptionkey@0"
I20260620 07:16:37.508736 12296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:37.523259 12296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:37.526115 12296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:37.526306 12296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:37.533419 12296 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:33607
I20260620 07:16:37.533711 12363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:33607 every 8 connection(s)
I20260620 07:16:37.534348 12296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:37.535223 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 12296
I20260620 07:16:37.535312 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:16:37.535574 11758 external_mini_cluster.cc:1428] Setting key 0ff49c165fca71afac2232d008a55383
I20260620 07:16:37.537400 12364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:37.544064 12364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7: Bootstrap starting.
Jun 20 07:16:37 dist-test-slave-j663 krb5kdc[12283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:37.546761 12364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:37.547627 12364 log.cc:824] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:37.549787 12364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7: No bootstrap required, opened a new log
I20260620 07:16:37.550118 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:37.536924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.11.123.190:33607)
0620 07:16:37.537357 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:37.537374 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:37.537417 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:16:37.538212 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:37.538234 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:37.538631 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:37.538972 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:37.539600 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:37.541107 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:37.541940 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:37.542632 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:37.542790 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:37.545206 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:37.545212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:37.545228 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:37.545261 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:37.547232 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:37.547647 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:37.547653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:37.547659 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:37.547790 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:37.548054 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:37.548058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:37.548059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:37.548496 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:37.548669 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:37.548927 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 07:16:37.549224 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":157,"threads_started":1}
I20260620 07:16:37.553017 12364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530ab13801e048578945e66a2fbec9a7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33607 } }
I20260620 07:16:37.553285 12364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:37.553334 12364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530ab13801e048578945e66a2fbec9a7, State: Initialized, Role: FOLLOWER
I20260620 07:16:37.553685 12364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530ab13801e048578945e66a2fbec9a7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33607 } }
I20260620 07:16:37.553794 12364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:37.553875 12364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:37.553974 12364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:37.554908 12364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530ab13801e048578945e66a2fbec9a7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33607 } }
I20260620 07:16:37.555235 12364 leader_election.cc:302] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 530ab13801e048578945e66a2fbec9a7; no voters:
I20260620 07:16:37.555557 12364 leader_election.cc:288] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:37.555768 12369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:37.556030 12369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [term 1 LEADER]: Becoming Leader. State: Replica: 530ab13801e048578945e66a2fbec9a7, State: Running, Role: LEADER
I20260620 07:16:37.556316 12369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530ab13801e048578945e66a2fbec9a7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33607 } }
I20260620 07:16:37.556695 12364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:37.557912 12370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "530ab13801e048578945e66a2fbec9a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530ab13801e048578945e66a2fbec9a7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33607 } } }
I20260620 07:16:37.558038 12370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:37.558519 12371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 530ab13801e048578945e66a2fbec9a7. Latest consensus state: current_term: 1 leader_uuid: "530ab13801e048578945e66a2fbec9a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530ab13801e048578945e66a2fbec9a7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33607 } } }
I20260620 07:16:37.558657 12371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:37.558511 12378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:37.561831 12378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:37.566874 12378 catalog_manager.cc:1347] Generated new cluster ID: 9d609a8c5ea0465c8d352fc83911e3cb
I20260620 07:16:37.566936 12378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:37.585568 12378 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:37.586551 12378 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:37.597105 12378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 530ab13801e048578945e66a2fbec9a7: Generated new TSK 0
I20260620 07:16:37.597795 12378 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:16:37.628098 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33607
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:37.744967 12390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:37.745281 12390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:37.745345 12390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:37.749279 12390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:37.749326 12390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:37.749382 12390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:16:37.754477 12390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33607
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:37.755368 12390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:37.756275 12390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:37.765709 12395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:37.765823 12390 server_base.cc:1034] running on GCE node
W20260620 07:16:37.765678 12396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:37.765681 12398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:37.766381 12390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:37.767019 12390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:37.768210 12390 hybrid_clock.cc:648] HybridClock initialized: now 1781939797768169 us; error 55 us; skew 500 ppm
Jun 20 07:16:37 dist-test-slave-j663 krb5kdc[12283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939797, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:37.770893 12390 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:16:37.771711 12390 webserver.cc:460] Webserver started at http://127.11.123.129:36969/ using document root <none> and password file <none>
I20260620 07:16:37.772270 12390 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:37.772360 12390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:37.772584 12390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:37.774571 12390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "c46078bad19c4123a559e39265a45eb4"
format_stamp: "Formatted at 2026-06-20 07:16:37 on dist-test-slave-j663"
server_key: "197bdf7ff610647ba3ec9e7f9f293758"
server_key_iv: "0fba1ac9adcfd99679bcb223126c4f7a"
server_key_version: "encryptionkey@0"
I20260620 07:16:37.775118 12390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "c46078bad19c4123a559e39265a45eb4"
format_stamp: "Formatted at 2026-06-20 07:16:37 on dist-test-slave-j663"
server_key: "197bdf7ff610647ba3ec9e7f9f293758"
server_key_iv: "0fba1ac9adcfd99679bcb223126c4f7a"
server_key_version: "encryptionkey@0"
I20260620 07:16:37.778754 12390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:16:37.781013 12405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:37.782199 12390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:37.782327 12390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "c46078bad19c4123a559e39265a45eb4"
format_stamp: "Formatted at 2026-06-20 07:16:37 on dist-test-slave-j663"
server_key: "197bdf7ff610647ba3ec9e7f9f293758"
server_key_iv: "0fba1ac9adcfd99679bcb223126c4f7a"
server_key_version: "encryptionkey@0"
I20260620 07:16:37.782442 12390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:37.806046 12390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:37.809165 12390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:37.809377 12390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:37.809962 12390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:37.810784 12390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:37.810853 12390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:37.810921 12390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:37.810972 12390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:37.819527 12390 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:44573
I20260620 07:16:37.819555 12518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:44573 every 8 connection(s)
I20260620 07:16:37.820430 12390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:16:37.824411 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 12390
I20260620 07:16:37.824503 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:16:37.824751 11758 external_mini_cluster.cc:1428] Setting key 3351f555dc3a4e5189c6b455b5031d72
Jun 20 07:16:37 dist-test-slave-j663 krb5kdc[12283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939797, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:16:37.834246 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:37.822246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:39663 (local address 127.11.123.190:33607)
0620 07:16:37.822378 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:37.822382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:37.823183 (+ 801us) server_negotiation.cc:407] Connection header received
0620 07:16:37.824089 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:37.824092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:37.824151 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:37.824244 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:37.825591 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:37.826199 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:37.827954 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:37.828188 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:37.828301 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:37.830747 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:37.830750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:37.830752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:37.830773 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:37.832197 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:37.832691 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:37.832701 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:37.832704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:37.832772 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:37.833141 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:37.833145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:37.833147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:37.833319 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:37.833425 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:37.833943 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 07:16:37.834076 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:16:37.835107 12521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:37.822627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33607 (local address 127.11.123.129:39663)
0620 07:16:37.823043 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:37.823089 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:37.823876 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:37.824411 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:37.824418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:37.824890 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:37.825452 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:37.825469 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:37.826316 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:37.826320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:37.827779 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:37.827795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:37.828336 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:37.828341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:37.828504 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:37.829164 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:37.829179 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:37.830576 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:37.832311 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:37.832317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:37.832331 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:37.832583 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:37.832882 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:37.832888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:37.832890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:37.833018 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:37.833422 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:37.833428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:37.833664 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 07:16:37.833890 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 07:16:37.834197 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
I20260620 07:16:37.836390 12519 heartbeater.cc:347] Connected to a master server at 127.11.123.190:33607
I20260620 07:16:37.836643 12519 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:37.837229 12519 heartbeater.cc:511] Master 127.11.123.190:33607 requested a full tablet report, sending...
I20260620 07:16:37.838796 12328 ts_manager.cc:194] Registered new tserver with Master: c46078bad19c4123a559e39265a45eb4 (127.11.123.129:44573)
I20260620 07:16:37.839980 12328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:39663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:16:37.850103 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33607
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:37.960057 12524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:37.960317 12524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:37.960387 12524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:37.964491 12524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:37.964547 12524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:37.964640 12524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:16:37.969789 12524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33607
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:37.970731 12524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:37.971670 12524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:37.981348 12529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:37.981444 12524 server_base.cc:1034] running on GCE node
W20260620 07:16:37.981634 12532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:37.981647 12530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:37.982105 12524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:37.982767 12524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:37.983955 12524 hybrid_clock.cc:648] HybridClock initialized: now 1781939797983919 us; error 47 us; skew 500 ppm
Jun 20 07:16:37 dist-test-slave-j663 krb5kdc[12283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939797, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:37.986764 12524 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:16:37.987607 12524 webserver.cc:460] Webserver started at http://127.11.123.130:34263/ using document root <none> and password file <none>
I20260620 07:16:37.988214 12524 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:37.988306 12524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:37.988535 12524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:37.990573 12524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "7544fc1b82c9421988eefabf4da28cc4"
format_stamp: "Formatted at 2026-06-20 07:16:37 on dist-test-slave-j663"
server_key: "27dc5f4429dc318922303ae1ebd867ea"
server_key_iv: "fc5833efb8063fef3087381771ad7463"
server_key_version: "encryptionkey@0"
I20260620 07:16:37.991129 12524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "7544fc1b82c9421988eefabf4da28cc4"
format_stamp: "Formatted at 2026-06-20 07:16:37 on dist-test-slave-j663"
server_key: "27dc5f4429dc318922303ae1ebd867ea"
server_key_iv: "fc5833efb8063fef3087381771ad7463"
server_key_version: "encryptionkey@0"
I20260620 07:16:37.994611 12524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 07:16:37.996840 12539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:37.998035 12524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:37.998162 12524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "7544fc1b82c9421988eefabf4da28cc4"
format_stamp: "Formatted at 2026-06-20 07:16:37 on dist-test-slave-j663"
server_key: "27dc5f4429dc318922303ae1ebd867ea"
server_key_iv: "fc5833efb8063fef3087381771ad7463"
server_key_version: "encryptionkey@0"
I20260620 07:16:37.998265 12524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:38.054597 12524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:38.058069 12524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:38.058256 12524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:38.058894 12524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:38.059706 12524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:38.059751 12524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:38.059788 12524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:38.059811 12524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:38.068590 12524 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:34975
I20260620 07:16:38.068616 12652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:34975 every 8 connection(s)
I20260620 07:16:38.069507 12524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:16:38.076274 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 12524
I20260620 07:16:38.076386 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:16:38.076889 11758 external_mini_cluster.cc:1428] Setting key 0df6756e03f61ba3081a10cbc1f24dc0
Jun 20 07:16:38 dist-test-slave-j663 krb5kdc[12283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939797, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:38.081861 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.071156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:59287 (local address 127.11.123.190:33607)
0620 07:16:38.071307 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:38.071312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:38.072156 (+ 844us) server_negotiation.cc:407] Connection header received
0620 07:16:38.072965 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:38.072968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:38.073018 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:38.073162 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:38.074202 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.074703 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.075733 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.075945 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.076036 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.078492 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:38.078495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:38.078501 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:38.078526 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:38.079906 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:38.080422 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:38.080427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:38.080429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:38.080509 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:38.080829 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:38.080833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:38.080835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:38.081009 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:38.081135 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:38.081592 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 07:16:38.081701 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:16:38.082569 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.071590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33607 (local address 127.11.123.130:59287)
0620 07:16:38.072004 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:38.072045 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:38.072782 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:38.073293 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:38.073299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:38.073639 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:38.074083 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.074095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.074803 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.074805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.075633 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.075641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.076081 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.076084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.076211 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.076747 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:38.076762 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:38.078339 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:38.080058 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:38.080064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:38.080077 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:38.080318 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:38.080604 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:38.080607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:38.080609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:38.080739 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:38.081146 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:38.081154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:38.081370 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 07:16:38.081550 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 07:16:38.081836 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
I20260620 07:16:38.083614 12653 heartbeater.cc:347] Connected to a master server at 127.11.123.190:33607
I20260620 07:16:38.083855 12653 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:38.084390 12653 heartbeater.cc:511] Master 127.11.123.190:33607 requested a full tablet report, sending...
I20260620 07:16:38.085471 12328 ts_manager.cc:194] Registered new tserver with Master: 7544fc1b82c9421988eefabf4da28cc4 (127.11.123.130:34975)
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
I20260620 07:16:38.086067 12328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:59287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:16:38.103327 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33607
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:38.210690 12658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:38.210945 12658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:38.211020 12658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:38.215083 12658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:38.215155 12658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:38.215240 12658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:16:38.220248 12658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33607
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:38.221230 12658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:38.222179 12658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:38.231357 12663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:38.231372 12664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:38.231426 12658 server_base.cc:1034] running on GCE node
W20260620 07:16:38.231372 12666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:38.231946 12658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:38.232607 12658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:38.233771 12658 hybrid_clock.cc:648] HybridClock initialized: now 1781939798233762 us; error 27 us; skew 500 ppm
Jun 20 07:16:38 dist-test-slave-j663 krb5kdc[12283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939798, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:38.236544 12658 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:16:38.237344 12658 webserver.cc:460] Webserver started at http://127.11.123.131:37987/ using document root <none> and password file <none>
I20260620 07:16:38.237859 12658 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:38.237913 12658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:38.238070 12658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:38.239969 12658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "66d17d02555f438985f2729d654d5a01"
format_stamp: "Formatted at 2026-06-20 07:16:38 on dist-test-slave-j663"
server_key: "8b983e2e5c97dae5a393514f6226075c"
server_key_iv: "8a6ccb033258c510f3d879ae46b8b496"
server_key_version: "encryptionkey@0"
I20260620 07:16:38.240473 12658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "66d17d02555f438985f2729d654d5a01"
format_stamp: "Formatted at 2026-06-20 07:16:38 on dist-test-slave-j663"
server_key: "8b983e2e5c97dae5a393514f6226075c"
server_key_iv: "8a6ccb033258c510f3d879ae46b8b496"
server_key_version: "encryptionkey@0"
I20260620 07:16:38.244010 12658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:16:38.246302 12673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:38.247593 12658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:16:38.247730 12658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "66d17d02555f438985f2729d654d5a01"
format_stamp: "Formatted at 2026-06-20 07:16:38 on dist-test-slave-j663"
server_key: "8b983e2e5c97dae5a393514f6226075c"
server_key_iv: "8a6ccb033258c510f3d879ae46b8b496"
server_key_version: "encryptionkey@0"
I20260620 07:16:38.247838 12658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:38.264060 12658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:38.267208 12658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:38.267351 12658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:38.267831 12658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:38.268573 12658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:38.268620 12658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:38.268657 12658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:38.268672 12658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:38.277194 12658 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:44799
I20260620 07:16:38.277275 12786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:44799 every 8 connection(s)
I20260620 07:16:38.278069 12658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:16:38.278684 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 12658
I20260620 07:16:38.278820 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:16:38.279079 11758 external_mini_cluster.cc:1428] Setting key a1b2140476bdf0cf89b97b65480c2d76
Jun 20 07:16:38 dist-test-slave-j663 krb5kdc[12283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939798, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:38.290951 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.280103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:35443 (local address 127.11.123.190:33607)
0620 07:16:38.280212 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:38.280215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:38.281480 (+ 1265us) server_negotiation.cc:407] Connection header received
0620 07:16:38.282400 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:38.282402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:38.282445 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:38.282517 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:38.283507 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.284015 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.285130 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.285339 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.285417 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.287704 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:38.287707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:38.287709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:38.287732 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:38.289029 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:38.289535 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:38.289540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:38.289542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:38.289621 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:38.289947 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:38.289951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:38.289953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:38.290138 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:38.290243 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:38.290683 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 07:16:38.290791 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 07:16:38.291750 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.280606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33607 (local address 127.11.123.131:35443)
0620 07:16:38.281322 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:38.281382 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:38.282237 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:38.282645 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:38.282651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:38.282968 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:38.283397 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.283408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.284113 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.284115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.284979 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.284987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.285457 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.285460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.285612 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.286138 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:38.286154 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:38.287545 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:38.289184 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:38.289190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:38.289202 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:38.289435 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:38.289718 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:38.289721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:38.289723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:38.289847 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:38.290239 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:38.290245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:38.290460 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 07:16:38.290643 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 07:16:38.290919 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":129,"threads_started":1}
I20260620 07:16:38.293052 12787 heartbeater.cc:347] Connected to a master server at 127.11.123.190:33607
I20260620 07:16:38.293293 12787 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:38.293751 12787 heartbeater.cc:511] Master 127.11.123.190:33607 requested a full tablet report, sending...
I20260620 07:16:38.294704 12327 ts_manager.cc:194] Registered new tserver with Master: 66d17d02555f438985f2729d654d5a01 (127.11.123.131:44799)
I20260620 07:16:38.295257 12327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:35443
I20260620 07:16:38.304646 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:16:38.314870 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.307853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.11.123.190:33607)
0620 07:16:38.307959 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:38.307963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:38.308099 (+ 136us) server_negotiation.cc:407] Connection header received
0620 07:16:38.308200 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:38.308203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:38.308235 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:38.308294 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:38.308964 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.309455 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.310263 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.310462 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.310536 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.311787 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:38.311790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:38.311792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:38.311810 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:38.313267 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:38.313805 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:38.313808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:38.313809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:38.313884 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:38.314173 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:38.314175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:38.314177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:38.314367 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:38.314472 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:38.314684 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 07:16:38.314766 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:16:38.319609 12327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:16:38.322635 12327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:16:38.336171 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.330615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:44573 (local address 127.0.0.1:38626)
0620 07:16:38.330888 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:38.330902 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:38.330983 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:38.331766 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:38.331769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:38.331787 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:38.331914 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.331923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.333485 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.333489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.334600 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.334607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.335578 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.335581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.335751 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.335791 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 07:16:38.335876 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:16:38.336014 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":13,"thread_start_us":101,"threads_started":1}
I20260620 07:16:38.336196 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.330211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:44799 (local address 127.0.0.1:47240)
0620 07:16:38.330552 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:38.330609 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:38.330729 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:38.331489 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:38.331497 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:38.331525 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:38.331691 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.331698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.333157 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.333161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.334475 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.334484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.335577 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.335581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.335764 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.335790 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 07:16:38.335886 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 07:16:38.336034 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":81,"thread_start_us":101,"threads_started":1}
I20260620 07:16:38.336481 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.331356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.11.123.130:34975)
0620 07:16:38.332057 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:38.332068 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:38.332088 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:16:38.332156 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:38.332160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:38.332335 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:38.332486 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:38.332681 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.333692 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.334978 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.335548 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.335664 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.335789 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:38.336238 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 07:16:38.336342 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":125,"threads_started":1}
I20260620 07:16:38.336630 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.330946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:34975 (local address 127.0.0.1:33838)
0620 07:16:38.331709 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:38.331722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:38.331799 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:38.332479 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:38.332482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:38.332501 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:38.332576 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.332581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.333816 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.333819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.334864 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.334872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.335860 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.335863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.335990 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.336024 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:16:38.336098 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 07:16:38.336154 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":27,"thread_start_us":82,"threads_started":1}
I20260620 07:16:38.336800 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.331015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.11.123.129:44573)
0620 07:16:38.331306 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:38.331320 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:38.331341 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:16:38.331422 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:38.331427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:38.331617 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:38.332058 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:38.332200 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.333346 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.334716 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.335441 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.336409 (+ 968us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.336531 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:38.336586 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 07:16:38.336666 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260620 07:16:38.339617 12453 tablet_service.cc:1467] Processing CreateTablet for tablet c08202f5acf141ab8da9b5111f019e70 (DEFAULT_TABLE table=test-table [id=575dd685a0b34500a2a12b5deb51d9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:38.339620 12587 tablet_service.cc:1467] Processing CreateTablet for tablet c08202f5acf141ab8da9b5111f019e70 (DEFAULT_TABLE table=test-table [id=575dd685a0b34500a2a12b5deb51d9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:38.340322 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.330413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.11.123.131:44799)
0620 07:16:38.331010 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:38.331020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:38.331042 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:16:38.331116 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:38.331123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:38.331320 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:38.331850 (+ 530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:38.332003 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.333008 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.334609 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.335455 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.335984 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.336111 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:38.336158 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 07:16:38.336772 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":100,"threads_started":1}
I20260620 07:16:38.340698 12587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c08202f5acf141ab8da9b5111f019e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:38.340699 12453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c08202f5acf141ab8da9b5111f019e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:38.342725 12721 tablet_service.cc:1467] Processing CreateTablet for tablet c08202f5acf141ab8da9b5111f019e70 (DEFAULT_TABLE table=test-table [id=575dd685a0b34500a2a12b5deb51d9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:38.343554 12721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c08202f5acf141ab8da9b5111f019e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:38.347677 12804 tablet_bootstrap.cc:492] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4: Bootstrap starting.
I20260620 07:16:38.348697 12805 tablet_bootstrap.cc:492] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4: Bootstrap starting.
I20260620 07:16:38.349494 12806 tablet_bootstrap.cc:492] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01: Bootstrap starting.
I20260620 07:16:38.349987 12804 tablet_bootstrap.cc:654] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:38.350698 12804 log.cc:824] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:38.351086 12805 tablet_bootstrap.cc:654] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:38.351594 12806 tablet_bootstrap.cc:654] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:38.351999 12805 log.cc:824] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:38.352715 12806 log.cc:824] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:38.352836 12804 tablet_bootstrap.cc:492] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4: No bootstrap required, opened a new log
I20260620 07:16:38.353016 12804 ts_tablet_manager.cc:1397] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:16:38.353915 12805 tablet_bootstrap.cc:492] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4: No bootstrap required, opened a new log
I20260620 07:16:38.354103 12805 ts_tablet_manager.cc:1397] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:16:38.354493 12806 tablet_bootstrap.cc:492] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01: No bootstrap required, opened a new log
I20260620 07:16:38.354689 12806 ts_tablet_manager.cc:1397] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:16:38.356684 12804 raft_consensus.cc:348] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d17d02555f438985f2729d654d5a01" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44799 } } peers { permanent_uuid: "7544fc1b82c9421988eefabf4da28cc4" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 34975 } } peers { permanent_uuid: "c46078bad19c4123a559e39265a45eb4" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44573 } }
I20260620 07:16:38.356914 12804 raft_consensus.cc:374] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:38.356988 12804 raft_consensus.cc:729] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46078bad19c4123a559e39265a45eb4, State: Initialized, Role: FOLLOWER
I20260620 07:16:38.357496 12804 consensus_queue.cc:260] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d17d02555f438985f2729d654d5a01" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44799 } } peers { permanent_uuid: "7544fc1b82c9421988eefabf4da28cc4" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 34975 } } peers { permanent_uuid: "c46078bad19c4123a559e39265a45eb4" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44573 } }
I20260620 07:16:38.357810 12805 raft_consensus.cc:348] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d17d02555f438985f2729d654d5a01" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44799 } } peers { permanent_uuid: "7544fc1b82c9421988eefabf4da28cc4" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 34975 } } peers { permanent_uuid: "c46078bad19c4123a559e39265a45eb4" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44573 } }
I20260620 07:16:38.358140 12805 raft_consensus.cc:374] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:38.358230 12805 raft_consensus.cc:729] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7544fc1b82c9421988eefabf4da28cc4, State: Initialized, Role: FOLLOWER
I20260620 07:16:38.358482 12804 ts_tablet_manager.cc:1428] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260620 07:16:38.358361 12806 raft_consensus.cc:348] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d17d02555f438985f2729d654d5a01" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44799 } } peers { permanent_uuid: "7544fc1b82c9421988eefabf4da28cc4" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 34975 } } peers { permanent_uuid: "c46078bad19c4123a559e39265a45eb4" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44573 } }
I20260620 07:16:38.358587 12806 raft_consensus.cc:374] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:38.358680 12806 raft_consensus.cc:729] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d17d02555f438985f2729d654d5a01, State: Initialized, Role: FOLLOWER
I20260620 07:16:38.358858 12519 heartbeater.cc:503] Master 127.11.123.190:33607 was elected leader, sending a full tablet report...
I20260620 07:16:38.359091 12806 consensus_queue.cc:260] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d17d02555f438985f2729d654d5a01" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44799 } } peers { permanent_uuid: "7544fc1b82c9421988eefabf4da28cc4" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 34975 } } peers { permanent_uuid: "c46078bad19c4123a559e39265a45eb4" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44573 } }
I20260620 07:16:38.359321 12805 consensus_queue.cc:260] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d17d02555f438985f2729d654d5a01" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44799 } } peers { permanent_uuid: "7544fc1b82c9421988eefabf4da28cc4" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 34975 } } peers { permanent_uuid: "c46078bad19c4123a559e39265a45eb4" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44573 } }
I20260620 07:16:38.359673 12787 heartbeater.cc:503] Master 127.11.123.190:33607 was elected leader, sending a full tablet report...
I20260620 07:16:38.360083 12653 heartbeater.cc:503] Master 127.11.123.190:33607 was elected leader, sending a full tablet report...
I20260620 07:16:38.360424 12805 ts_tablet_manager.cc:1428] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 07:16:38.361064 12806 ts_tablet_manager.cc:1428] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 07:16:38.529946 12788 tablet.cc:2369] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:16:38.570557 12654 tablet.cc:2369] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:16:38.571705 12520 tablet.cc:2369] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:16:38.587111 12810 raft_consensus.cc:482] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:16:38.587344 12810 raft_consensus.cc:504] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d17d02555f438985f2729d654d5a01" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44799 } } peers { permanent_uuid: "7544fc1b82c9421988eefabf4da28cc4" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 34975 } } peers { permanent_uuid: "c46078bad19c4123a559e39265a45eb4" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44573 } }
I20260620 07:16:38.588538 12810 leader_election.cc:288] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66d17d02555f438985f2729d654d5a01 (127.11.123.131:44799), 7544fc1b82c9421988eefabf4da28cc4 (127.11.123.130:34975)
I20260620 07:16:38.592334 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.588866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:56787 (local address 127.11.123.131:44799)
0620 07:16:38.589045 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:38.589050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:38.589199 (+ 149us) server_negotiation.cc:407] Connection header received
0620 07:16:38.589379 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:38.589383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:38.589436 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:38.589528 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:38.589841 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.590344 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.591403 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.591801 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.591925 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.591975 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:38.592155 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 07:16:38.592216 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:16:38.592350 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.588756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:44799 (local address 127.11.123.129:56787)
0620 07:16:38.589040 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:38.589059 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:38.589248 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:38.589565 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:38.589569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:38.589591 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:38.589734 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.589743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.590459 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.590463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.591274 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.591282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.591920 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.591924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.592019 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.592048 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 07:16:38.592139 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:16:38.592207 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":94,"threads_started":1}
I20260620 07:16:38.592708 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.589322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:37275 (local address 127.11.123.130:34975)
0620 07:16:38.589437 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:38.589443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:38.589651 (+ 208us) server_negotiation.cc:407] Connection header received
0620 07:16:38.589782 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:38.589784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:38.589825 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:38.589912 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:38.590265 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.590754 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.591747 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.592130 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.592253 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.592297 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:38.592518 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 07:16:38.592597 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 07:16:38.593025 12814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.589222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:34975 (local address 127.11.123.129:37275)
0620 07:16:38.589576 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:38.589588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:38.589686 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:38.589923 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:38.589926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:38.589944 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:38.590151 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.590158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.590865 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.590869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.591627 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:38.591635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.592247 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.592251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:38.592347 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.592373 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 07:16:38.592849 (+ 476us) client_negotiation.cc:240] Negotiation successful
0620 07:16:38.592921 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":123,"threads_started":1}
I20260620 07:16:38.592971 12741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08202f5acf141ab8da9b5111f019e70" candidate_uuid: "c46078bad19c4123a559e39265a45eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d17d02555f438985f2729d654d5a01" is_pre_election: true
I20260620 07:16:38.593317 12741 raft_consensus.cc:2454] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c46078bad19c4123a559e39265a45eb4 in term 0.
I20260620 07:16:38.593690 12607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08202f5acf141ab8da9b5111f019e70" candidate_uuid: "c46078bad19c4123a559e39265a45eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7544fc1b82c9421988eefabf4da28cc4" is_pre_election: true
I20260620 07:16:38.593813 12408 leader_election.cc:302] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66d17d02555f438985f2729d654d5a01, c46078bad19c4123a559e39265a45eb4; no voters:
I20260620 07:16:38.593951 12607 raft_consensus.cc:2454] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c46078bad19c4123a559e39265a45eb4 in term 0.
I20260620 07:16:38.594084 12810 raft_consensus.cc:2781] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:16:38.594178 12810 raft_consensus.cc:482] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:16:38.594240 12810 raft_consensus.cc:3037] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:38.595257 12810 raft_consensus.cc:504] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d17d02555f438985f2729d654d5a01" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44799 } } peers { permanent_uuid: "7544fc1b82c9421988eefabf4da28cc4" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 34975 } } peers { permanent_uuid: "c46078bad19c4123a559e39265a45eb4" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44573 } }
I20260620 07:16:38.595566 12810 leader_election.cc:288] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [CANDIDATE]: Term 1 election: Requested vote from peers 66d17d02555f438985f2729d654d5a01 (127.11.123.131:44799), 7544fc1b82c9421988eefabf4da28cc4 (127.11.123.130:34975)
I20260620 07:16:38.595907 12607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08202f5acf141ab8da9b5111f019e70" candidate_uuid: "c46078bad19c4123a559e39265a45eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7544fc1b82c9421988eefabf4da28cc4"
I20260620 07:16:38.595911 12741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08202f5acf141ab8da9b5111f019e70" candidate_uuid: "c46078bad19c4123a559e39265a45eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d17d02555f438985f2729d654d5a01"
I20260620 07:16:38.596040 12607 raft_consensus.cc:3037] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:38.596060 12741 raft_consensus.cc:3037] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:38.597003 12741 raft_consensus.cc:2454] T c08202f5acf141ab8da9b5111f019e70 P 66d17d02555f438985f2729d654d5a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c46078bad19c4123a559e39265a45eb4 in term 1.
I20260620 07:16:38.597025 12607 raft_consensus.cc:2454] T c08202f5acf141ab8da9b5111f019e70 P 7544fc1b82c9421988eefabf4da28cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c46078bad19c4123a559e39265a45eb4 in term 1.
I20260620 07:16:38.597306 12408 leader_election.cc:302] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7544fc1b82c9421988eefabf4da28cc4, c46078bad19c4123a559e39265a45eb4; no voters:
I20260620 07:16:38.597507 12810 raft_consensus.cc:2781] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:38.597718 12810 raft_consensus.cc:686] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [term 1 LEADER]: Becoming Leader. State: Replica: c46078bad19c4123a559e39265a45eb4, State: Running, Role: LEADER
I20260620 07:16:38.598031 12810 consensus_queue.cc:237] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d17d02555f438985f2729d654d5a01" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44799 } } peers { permanent_uuid: "7544fc1b82c9421988eefabf4da28cc4" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 34975 } } peers { permanent_uuid: "c46078bad19c4123a559e39265a45eb4" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44573 } }
I20260620 07:16:38.601361 12326 catalog_manager.cc:5495] T c08202f5acf141ab8da9b5111f019e70 P c46078bad19c4123a559e39265a45eb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c46078bad19c4123a559e39265a45eb4 (127.11.123.129). New cstate: current_term: 1 leader_uuid: "c46078bad19c4123a559e39265a45eb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d17d02555f438985f2729d654d5a01" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7544fc1b82c9421988eefabf4da28cc4" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c46078bad19c4123a559e39265a45eb4" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44573 } health_report { overall_health: HEALTHY } } }
Jun 20 07:16:38 dist-test-slave-j663 krb5kdc[12283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.129@KRBTEST.COM
I20260620 07:16:38.628176 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.619527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.11.123.129:44573)
0620 07:16:38.619687 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:38.619691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:38.619759 (+ 68us) server_negotiation.cc:407] Connection header received
0620 07:16:38.619873 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:38.619876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:38.619915 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:38.619985 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:38.620709 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.621214 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.622017 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.622214 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.622287 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.624674 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:38.624684 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:38.624697 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:38.624718 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:38.626445 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:38.626894 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:38.626900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:38.626901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:38.626994 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:38.627269 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:38.627273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:38.627275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:38.627521 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:38.627734 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:38.627885 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 07:16:38.628009 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 07:16:38 dist-test-slave-j663 krb5kdc[12283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939798, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:16:38 dist-test-slave-j663 krb5kdc[12283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939798, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.123.129@KRBTEST.COM
I20260620 07:16:38.654094 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.646044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.11.123.129:44573)
0620 07:16:38.646176 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:38.646182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:38.646349 (+ 167us) server_negotiation.cc:407] Connection header received
0620 07:16:38.646541 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:38.646544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:38.646583 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:38.646672 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:38.647357 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.647869 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.648689 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.648917 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.649027 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.651399 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:38.651402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:38.651404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:38.651424 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:38.652692 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:38.653039 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:38.653041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:38.653042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:38.653160 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:38.653405 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:38.653408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:38.653409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:38.653575 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:38.653687 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:38.653895 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 07:16:38.653980 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 07:16:38 dist-test-slave-j663 krb5kdc[12283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:16:38 dist-test-slave-j663 krb5kdc[12283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.129@KRBTEST.COM
I20260620 07:16:38.678049 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:38.670031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.11.123.129:44573)
0620 07:16:38.670160 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:38.670164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:38.670321 (+ 157us) server_negotiation.cc:407] Connection header received
0620 07:16:38.670412 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:38.670414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:38.670450 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:38.670515 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:38.671101 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.671579 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.672331 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:38.672524 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:38.672612 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:38.675200 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:38.675203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:38.675205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:38.675225 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:38.676630 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:38.676980 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:38.676982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:38.676983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:38.677049 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:38.677313 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:38.677315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:38.677316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:38.677481 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:38.677613 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:38.677751 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 07:16:38.677836 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 07:16:38.679319 12433 tablet_service.cc:2908] Rejecting scan request for tablet c08202f5acf141ab8da9b5111f019e70: Uninitialized: clean time has not yet been initialized
I20260620 07:16:38.681633 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 12390
I20260620 07:16:38.687901 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 12524
I20260620 07:16:38.693169 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 12658
I20260620 07:16:38.698093 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 12296
2026-06-20T07:16:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2510 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:38 dist-test-slave-j663 krb5kdc[12829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:38 dist-test-slave-j663 krb5kdc[12829](info): set up 2 sockets
Jun 20 07:16:38 dist-test-slave-j663 krb5kdc[12829](info): commencing operation
krb5kdc: starting...
W20260620 07:16:39.740875 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:16:39 dist-test-slave-j663 krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:16:39.863444 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:36289
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:36289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:39.969139 12842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:39.969380 12842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:39.969419 12842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:39.973294 12842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:39.973343 12842 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 07:16:39.973361 12842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:39.973378 12842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:39.973394 12842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:39.973409 12842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:39.978427 12842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:36289
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:36289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:39.979354 12842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:39.980271 12842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:39.988540 12850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:39.988581 12848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:39.988644 12842 server_base.cc:1034] running on GCE node
W20260620 07:16:39.988583 12847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:39.989449 12842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:39.990571 12842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:39.991750 12842 hybrid_clock.cc:648] HybridClock initialized: now 1781939799991734 us; error 29 us; skew 500 ppm
Jun 20 07:16:39 dist-test-slave-j663 krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939799, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:39.994573 12842 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:16:39.995441 12842 webserver.cc:460] Webserver started at http://127.11.123.190:40751/ using document root <none> and password file <none>
I20260620 07:16:39.996021 12842 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:39.996112 12842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:39.996335 12842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:39.998509 12842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "76ee2a3b32344b20a13c60e056e57b20"
format_stamp: "Formatted at 2026-06-20 07:16:39 on dist-test-slave-j663"
server_key: "b8071791a4e6b3c3fa47789de7e9ba08"
server_key_iv: "40446d58b6c66cec4d587df126acb22e"
server_key_version: "encryptionkey@0"
I20260620 07:16:39.999086 12842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "76ee2a3b32344b20a13c60e056e57b20"
format_stamp: "Formatted at 2026-06-20 07:16:39 on dist-test-slave-j663"
server_key: "b8071791a4e6b3c3fa47789de7e9ba08"
server_key_iv: "40446d58b6c66cec4d587df126acb22e"
server_key_version: "encryptionkey@0"
I20260620 07:16:40.002797 12842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:16:40.005241 12857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:40.006412 12842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:40.006552 12842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "76ee2a3b32344b20a13c60e056e57b20"
format_stamp: "Formatted at 2026-06-20 07:16:39 on dist-test-slave-j663"
server_key: "b8071791a4e6b3c3fa47789de7e9ba08"
server_key_iv: "40446d58b6c66cec4d587df126acb22e"
server_key_version: "encryptionkey@0"
I20260620 07:16:40.006659 12842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:40.032088 12842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:40.034801 12842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:40.034996 12842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:40.042586 12842 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:36289
I20260620 07:16:40.042598 12919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:36289 every 8 connection(s)
I20260620 07:16:40.043454 12842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:40.046094 12920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:40.048940 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 12842
I20260620 07:16:40.049060 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:16:40.049353 11758 external_mini_cluster.cc:1428] Setting key 922d3dbb8ecc99e9d06d52b7cdc39022
I20260620 07:16:40.051687 12920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20: Bootstrap starting.
I20260620 07:16:40.054051 12920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:40.054744 12920 log.cc:824] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:40.056581 12920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20: No bootstrap required, opened a new log
Jun 20 07:16:40 dist-test-slave-j663 krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:40.061136 12920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ee2a3b32344b20a13c60e056e57b20" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36289 } }
I20260620 07:16:40.061419 12920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:40.061535 12920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ee2a3b32344b20a13c60e056e57b20, State: Initialized, Role: FOLLOWER
I20260620 07:16:40.062049 12920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ee2a3b32344b20a13c60e056e57b20" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36289 } }
I20260620 07:16:40.062206 12920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:40.062299 12920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:40.062417 12920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:40.063467 12920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ee2a3b32344b20a13c60e056e57b20" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36289 } }
I20260620 07:16:40.063642 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.050615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.11.123.190:36289)
0620 07:16:40.050989 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.051003 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.051037 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:16:40.051615 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.051629 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.051988 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.052245 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:40.053136 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.054128 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.054956 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.055543 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.055692 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.058420 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:40.058427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:40.058442 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:40.058473 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:40.060561 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.061027 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.061033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.061039 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.061210 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.061509 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.061513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.061516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.062030 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:40.062252 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.062529 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.062843 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
I20260620 07:16:40.063890 12920 leader_election.cc:302] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76ee2a3b32344b20a13c60e056e57b20; no voters:
I20260620 07:16:40.064244 12920 leader_election.cc:288] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:40.064321 12925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:40.064505 12925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [term 1 LEADER]: Becoming Leader. State: Replica: 76ee2a3b32344b20a13c60e056e57b20, State: Running, Role: LEADER
I20260620 07:16:40.064882 12925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ee2a3b32344b20a13c60e056e57b20" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36289 } }
I20260620 07:16:40.066717 12927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76ee2a3b32344b20a13c60e056e57b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ee2a3b32344b20a13c60e056e57b20" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36289 } } }
I20260620 07:16:40.066753 12926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76ee2a3b32344b20a13c60e056e57b20. Latest consensus state: current_term: 1 leader_uuid: "76ee2a3b32344b20a13c60e056e57b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ee2a3b32344b20a13c60e056e57b20" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36289 } } }
I20260620 07:16:40.066867 12920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:40.066948 12927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:40.066972 12926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:40.067448 12929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:40.070456 12929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:40.076190 12929 catalog_manager.cc:1347] Generated new cluster ID: c4ede0909e4b49cd8dcb210cc0ad6ca5
I20260620 07:16:40.076256 12929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:40.084535 12929 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:40.085469 12929 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:40.089313 12929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20: Generated new TSK 0
I20260620 07:16:40.089927 12929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:16:40.120868 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36289
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:40.229395 12946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:40.229627 12946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:40.229667 12946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:40.233611 12946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:40.233661 12946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:40.233719 12946 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:16:40.233799 12946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:16:40.238858 12946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.123.190:36289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:40.239913 12946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:40.240926 12946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:40.250103 12952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:40.250108 12951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:40.250212 12946 server_base.cc:1034] running on GCE node
W20260620 07:16:40.250103 12954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:40.250792 12946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:40.251422 12946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:40.252614 12946 hybrid_clock.cc:648] HybridClock initialized: now 1781939800252597 us; error 29 us; skew 500 ppm
Jun 20 07:16:40 dist-test-slave-j663 krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939800, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:40.255280 12946 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:16:40.256049 12946 webserver.cc:460] Webserver started at http://127.11.123.129:38319/ using document root <none> and password file <none>
I20260620 07:16:40.256536 12946 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:40.256592 12946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:40.256742 12946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:40.258735 12946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "db88f9ca298d4eac82a64b81c514c530"
format_stamp: "Formatted at 2026-06-20 07:16:40 on dist-test-slave-j663"
server_key: "5d36889de7d0cd544e7860e7c0d07918"
server_key_iv: "796c6a0306d692c8add9d0685dbc8b9e"
server_key_version: "encryptionkey@0"
I20260620 07:16:40.259238 12946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "db88f9ca298d4eac82a64b81c514c530"
format_stamp: "Formatted at 2026-06-20 07:16:40 on dist-test-slave-j663"
server_key: "5d36889de7d0cd544e7860e7c0d07918"
server_key_iv: "796c6a0306d692c8add9d0685dbc8b9e"
server_key_version: "encryptionkey@0"
I20260620 07:16:40.262722 12946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 07:16:40.264937 12961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:40.266125 12946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:40.266215 12946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "db88f9ca298d4eac82a64b81c514c530"
format_stamp: "Formatted at 2026-06-20 07:16:40 on dist-test-slave-j663"
server_key: "5d36889de7d0cd544e7860e7c0d07918"
server_key_iv: "796c6a0306d692c8add9d0685dbc8b9e"
server_key_version: "encryptionkey@0"
I20260620 07:16:40.266320 12946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:40.285785 12946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:40.288868 12946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:40.289064 12946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:40.290486 12946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:40.290557 12946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:40.290627 12946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:40.290673 12946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:16:40 dist-test-slave-j663 krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939800, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:40.301008 12946 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:42805
I20260620 07:16:40.301124 13076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:42805 every 8 connection(s)
I20260620 07:16:40.301951 12946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:16:40.305249 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.291342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:55143 (local address 127.11.123.190:36289)
0620 07:16:40.291494 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.291497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.292522 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 07:16:40.293604 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.293610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.293688 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.293753 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:40.295547 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.296458 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.298179 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.298568 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.298689 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.301548 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:40.301551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:40.301553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:40.301577 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:40.303166 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.303739 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.303742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.303744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.303832 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.304158 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.304162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.304163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.304346 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:40.304449 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.304936 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.305056 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:16:40.305979 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.291902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36289 (local address 127.11.123.129:55143)
0620 07:16:40.292351 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:40.292390 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:40.293343 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:40.294228 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:40.294238 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:40.294667 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:40.295323 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.295345 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.296595 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.296599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.298061 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.298071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.298740 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.298743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.298977 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.299633 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:40.299648 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:40.301388 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:40.303279 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:40.303287 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:40.303303 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:40.303632 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:40.303921 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:40.303925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:40.303927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:40.304061 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:40.304446 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:40.304452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:40.304687 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 07:16:40.304877 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 07:16:40.305203 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":121,"threads_started":1}
I20260620 07:16:40.307090 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 12946
I20260620 07:16:40.307207 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:16:40.307484 11758 external_mini_cluster.cc:1428] Setting key 771ca2b7cdfae77e64524acdeafa5332
I20260620 07:16:40.309252 13077 heartbeater.cc:347] Connected to a master server at 127.11.123.190:36289
I20260620 07:16:40.309393 13077 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:40.309852 13077 heartbeater.cc:511] Master 127.11.123.190:36289 requested a full tablet report, sending...
I20260620 07:16:40.311280 12874 ts_manager.cc:194] Registered new tserver with Master: db88f9ca298d4eac82a64b81c514c530 (127.11.123.129:42805)
I20260620 07:16:40.312795 12874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:55143
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:16:40.317934 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.309626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36289 (local address 127.11.123.129:54197)
0620 07:16:40.309912 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:40.309929 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:40.310064 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:40.310394 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:40.310397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:40.310634 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:40.310753 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.310760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.311809 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.311812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.312683 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.312693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.313350 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.313357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.313466 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.314027 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:40.314041 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:40.314466 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:40.316466 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:40.316469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:40.316471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:40.316660 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:40.316959 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:40.316961 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:40.316962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:40.317036 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:40.317527 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:40.317529 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:40.317600 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 07:16:40.317680 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:16:40.317777 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
I20260620 07:16:40.317943 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.309445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:54197 (local address 127.11.123.190:36289)
0620 07:16:40.309549 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.309553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.309995 (+ 442us) server_negotiation.cc:407] Connection header received
0620 07:16:40.310166 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.310169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.310211 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.310291 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:40.310859 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.311668 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.312806 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.313226 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.313348 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.314609 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:40.314614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:40.314617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:40.314648 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:40.316334 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.316762 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.316766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.316767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.316861 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.317184 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.317189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.317190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.317412 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:40.317516 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.317745 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.317842 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:16:40.335583 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36289
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:40.445925 13086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:40.446197 13086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:40.446241 13086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:40.450258 13086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:40.450312 13086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:40.450385 13086 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:16:40.450465 13086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:16:40.455617 13086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.123.190:36289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:40.456650 13086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:40.457684 13086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:40.467038 13091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:40.467226 13086 server_base.cc:1034] running on GCE node
W20260620 07:16:40.467056 13094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:40.467054 13092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:40.467762 13086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:40.468390 13086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:40.469532 13086 hybrid_clock.cc:648] HybridClock initialized: now 1781939800469518 us; error 24 us; skew 500 ppm
Jun 20 07:16:40 dist-test-slave-j663 krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939800, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:40.472147 13086 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:16:40.472951 13086 webserver.cc:460] Webserver started at http://127.11.123.130:36829/ using document root <none> and password file <none>
I20260620 07:16:40.473505 13086 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:40.473572 13086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:40.473733 13086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:40.475627 13086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13"
format_stamp: "Formatted at 2026-06-20 07:16:40 on dist-test-slave-j663"
server_key: "b09df61c11a19c5c1d305134d2625fd1"
server_key_iv: "ddd4d0e1c19623cb44aba095295d21c8"
server_key_version: "encryptionkey@0"
I20260620 07:16:40.476130 13086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13"
format_stamp: "Formatted at 2026-06-20 07:16:40 on dist-test-slave-j663"
server_key: "b09df61c11a19c5c1d305134d2625fd1"
server_key_iv: "ddd4d0e1c19623cb44aba095295d21c8"
server_key_version: "encryptionkey@0"
I20260620 07:16:40.479643 13086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 07:16:40.481949 13101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:40.483079 13086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:16:40.483170 13086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13"
format_stamp: "Formatted at 2026-06-20 07:16:40 on dist-test-slave-j663"
server_key: "b09df61c11a19c5c1d305134d2625fd1"
server_key_iv: "ddd4d0e1c19623cb44aba095295d21c8"
server_key_version: "encryptionkey@0"
I20260620 07:16:40.483249 13086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:40.499415 13086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:40.502223 13086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:40.502369 13086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:40.503677 13086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:40.503748 13086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:40.503818 13086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:40.503858 13086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:16:40 dist-test-slave-j663 krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939800, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:40.514392 13086 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:37005
I20260620 07:16:40.514608 13216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:37005 every 8 connection(s)
I20260620 07:16:40.515372 13086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:16:40.517284 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.504375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:33309 (local address 127.11.123.190:36289)
0620 07:16:40.504530 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.504536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.505846 (+ 1310us) server_negotiation.cc:407] Connection header received
0620 07:16:40.507063 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.507066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.507125 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.507225 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:40.508528 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.509132 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.510404 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.510737 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.510847 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.513548 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:40.513552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:40.513554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:40.513584 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:40.515047 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.515612 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.515615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.515616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.515688 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.516139 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.516142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.516146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.516354 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:40.516437 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.516992 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.517159 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:16:40.517982 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.505290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36289 (local address 127.11.123.130:33309)
0620 07:16:40.505708 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:40.505754 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:40.506790 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:40.507390 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:40.507404 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:40.507894 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:40.508384 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.508413 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.509251 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.509255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.510270 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.510311 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.510890 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.510894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.511083 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.511645 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:40.511660 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:40.513374 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:40.515157 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:40.515162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:40.515175 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:40.515508 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:40.515802 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:40.515805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:40.515807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:40.515969 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:40.516532 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:40.516540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:40.516772 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 07:16:40.516946 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 07:16:40.517235 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":122,"threads_started":1}
I20260620 07:16:40.519064 13217 heartbeater.cc:347] Connected to a master server at 127.11.123.190:36289
I20260620 07:16:40.519277 13217 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:40.519869 13217 heartbeater.cc:511] Master 127.11.123.190:36289 requested a full tablet report, sending...
I20260620 07:16:40.520927 12873 ts_manager.cc:194] Registered new tserver with Master: db5fdbaf614e46c0ba582fbe3d2e1d13 (127.11.123.130:37005)
I20260620 07:16:40.521314 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 13086
I20260620 07:16:40.521430 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:16:40.521667 12873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:33309
I20260620 07:16:40.521727 11758 external_mini_cluster.cc:1428] Setting key 9ab7dc363b8bb676371a7b1ef84875fb
I20260620 07:16:40.529477 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.521948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:34759 (local address 127.11.123.190:36289)
0620 07:16:40.522069 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.522072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.522084 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:16:40.522116 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.522122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.522161 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.522236 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:40.523978 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.524448 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.525264 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.525468 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.525571 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.526594 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:40.526598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:40.526600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:40.526627 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:40.527910 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.528308 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.528312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.528314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.528386 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.528650 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.528654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.528656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.528844 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:40.528947 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.529238 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.529336 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:16:40.529496 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.521407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36289 (local address 127.11.123.130:34759)
0620 07:16:40.521682 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:40.521699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:40.521780 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:40.523621 (+ 1841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:40.523623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:40.523776 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:40.523858 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.523863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.524568 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.524571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.525159 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.525167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.525572 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.525575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.525661 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.526148 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:40.526159 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:40.526439 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:40.528025 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:40.528031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:40.528034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:40.528216 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:40.528478 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:40.528481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:40.528483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:40.528565 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:40.528937 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:40.528940 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:40.529023 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 07:16:40.529149 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 07:16:40.529289 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:16:40.547447 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36289
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:40.654800 13226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:40.655035 13226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:40.655078 13226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:40.658893 13226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:40.658941 13226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:40.658977 13226 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:16:40.659008 13226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:16:40.664011 13226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.123.190:36289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:40.664901 13226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:40.665822 13226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:40.675196 13232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:40.675163 13231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:40.675210 13226 server_base.cc:1034] running on GCE node
W20260620 07:16:40.675163 13234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:40.676091 13226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:40.676671 13226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:40.677853 13226 hybrid_clock.cc:648] HybridClock initialized: now 1781939800677834 us; error 31 us; skew 500 ppm
Jun 20 07:16:40 dist-test-slave-j663 krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939800, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:40.680519 13226 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:16:40.681424 13226 webserver.cc:460] Webserver started at http://127.11.123.131:37873/ using document root <none> and password file <none>
I20260620 07:16:40.681962 13226 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:40.682070 13226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:40.682343 13226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:40.684278 13226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "de5f98910cee49b2b4d4e9591dab9fb6"
format_stamp: "Formatted at 2026-06-20 07:16:40 on dist-test-slave-j663"
server_key: "92cdd7a2d527d84b06c7e095b33e90a2"
server_key_iv: "f42aaaef627209a126d984f6158a51e5"
server_key_version: "encryptionkey@0"
I20260620 07:16:40.684808 13226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "de5f98910cee49b2b4d4e9591dab9fb6"
format_stamp: "Formatted at 2026-06-20 07:16:40 on dist-test-slave-j663"
server_key: "92cdd7a2d527d84b06c7e095b33e90a2"
server_key_iv: "f42aaaef627209a126d984f6158a51e5"
server_key_version: "encryptionkey@0"
I20260620 07:16:40.688498 13226 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:16:40.690816 13241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:40.691960 13226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 07:16:40.692094 13226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "de5f98910cee49b2b4d4e9591dab9fb6"
format_stamp: "Formatted at 2026-06-20 07:16:40 on dist-test-slave-j663"
server_key: "92cdd7a2d527d84b06c7e095b33e90a2"
server_key_iv: "f42aaaef627209a126d984f6158a51e5"
server_key_version: "encryptionkey@0"
I20260620 07:16:40.692199 13226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:40.700506 13226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:40.703442 13226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:40.703622 13226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:40.704921 13226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:40.704985 13226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:40.705053 13226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:40.705113 13226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:16:40 dist-test-slave-j663 krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939800, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:40.715703 13226 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:34943
I20260620 07:16:40.716368 13356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:34943 every 8 connection(s)
I20260620 07:16:40.716564 13226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:16:40.718107 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.705644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:34443 (local address 127.11.123.190:36289)
0620 07:16:40.705800 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.705807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.706947 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 07:16:40.708038 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.708044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.708108 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.708216 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:40.709467 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.710158 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.711553 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.711857 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.711978 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.714736 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:40.714741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:40.714744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:40.714775 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:40.716209 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.716748 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.716751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.716752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.716822 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.717229 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.717234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.717239 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.717440 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:40.717516 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.717905 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.717994 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:16:40.718827 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.706385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36289 (local address 127.11.123.131:34443)
0620 07:16:40.706787 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:40.706860 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:40.707837 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:40.708411 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:40.708417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:40.708763 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:40.709324 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.709336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.710276 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.710280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.711380 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.711389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.712020 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.712024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.712207 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.712709 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:40.712723 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:40.714560 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:40.716315 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:40.716320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:40.716332 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:40.716661 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:40.716923 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:40.716928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:40.716931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:40.717107 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:40.717524 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:40.717529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:40.717714 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 07:16:40.717848 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 07:16:40.718069 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":112,"threads_started":1}
I20260620 07:16:40.719889 13357 heartbeater.cc:347] Connected to a master server at 127.11.123.190:36289
I20260620 07:16:40.720027 13357 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:40.720548 13357 heartbeater.cc:511] Master 127.11.123.190:36289 requested a full tablet report, sending...
I20260620 07:16:40.721557 12874 ts_manager.cc:194] Registered new tserver with Master: de5f98910cee49b2b4d4e9591dab9fb6 (127.11.123.131:34943)
I20260620 07:16:40.722297 12874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:34443
I20260620 07:16:40.722914 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 13226
I20260620 07:16:40.723023 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:16:40.723263 11758 external_mini_cluster.cc:1428] Setting key b8e7fd88ff0df2612cedcabf9914ba88
I20260620 07:16:40.724992 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:16:40.729174 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.722073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:33461 (local address 127.11.123.190:36289)
0620 07:16:40.722179 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.722182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.722376 (+ 194us) server_negotiation.cc:407] Connection header received
0620 07:16:40.722477 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.722480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.722522 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.722628 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:40.723095 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.723815 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.724684 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.724999 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.725136 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.726224 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:40.726226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:40.726228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:40.726247 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:40.727602 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.728077 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.728079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.728081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.728144 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.728414 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.728416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.728417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.728577 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:40.728649 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.728938 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.729017 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 07:16:40.729321 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.722001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36289 (local address 127.11.123.131:33461)
0620 07:16:40.722293 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:40.722304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:40.722400 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:40.722673 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:40.722675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:40.722849 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:40.722962 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.722970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.723968 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.723971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.724586 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.724594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.725144 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.725148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.725232 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.725658 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:40.725675 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:40.726102 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:40.727702 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:40.727707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:40.727709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:40.727982 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:40.728223 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:40.728228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:40.728233 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:40.728330 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:40.728667 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:40.728672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:40.728776 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 07:16:40.728892 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 07:16:40.729058 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":83,"threads_started":1}
Jun 20 07:16:40 dist-test-slave-j663 krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939800, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:16:40 dist-test-slave-j663 krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939800, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:40.750527 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.742631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.11.123.190:36289)
0620 07:16:40.742743 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.742747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.742757 (+ 10us) server_negotiation.cc:407] Connection header received
0620 07:16:40.742793 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.742795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.742828 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.742887 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:40.743547 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.744030 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.744773 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.744962 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.745032 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.747411 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:40.747414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:40.747416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:40.747436 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:40.748793 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.749249 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.749252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.749254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.749332 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:40.749721 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:40.749724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:40.749725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:40.749896 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:40.749998 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.750197 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.750288 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 07:16:40.753593 12874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:16:40.757266 12874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:16:40.771733 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.765688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:34943 (local address 127.0.0.1:53606)
0620 07:16:40.766184 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:40.766230 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:40.766359 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:40.767118 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:40.767123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:40.767145 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:40.767290 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.767298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.768756 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.768760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.770256 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.770265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.771194 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.771197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.771306 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.771346 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 07:16:40.771455 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 07:16:40.771568 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260620 07:16:40.771759 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.765820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53606 (local address 127.11.123.131:34943)
0620 07:16:40.766249 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.766263 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.766296 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:16:40.766568 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.766574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.766785 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.766947 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:40.767414 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.768656 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.770369 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.771073 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.771219 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.771347 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.771507 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.771602 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":110,"threads_started":1}
I20260620 07:16:40.772425 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.765843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:42805 (local address 127.0.0.1:42246)
0620 07:16:40.767126 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:40.767142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:40.767238 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:40.767750 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:40.767753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:40.767770 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:40.767876 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.767886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.769154 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.769174 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.770799 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.770805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.771984 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.771988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.772069 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.772089 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:16:40.772165 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 07:16:40.772232 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"mutex_wait_us":72,"thread_start_us":86,"threads_started":1}
I20260620 07:16:40.772439 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.766599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.11.123.129:42805)
0620 07:16:40.766966 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.766980 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.767351 (+ 371us) server_negotiation.cc:407] Connection header received
0620 07:16:40.767424 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.767427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.767616 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.767778 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:40.768008 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.769007 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.770920 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.771638 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.771765 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.771922 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.772215 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.772300 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":82,"threads_started":1}
I20260620 07:16:40.772729 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.766597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.11.123.130:37005)
0620 07:16:40.766966 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.766979 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.767699 (+ 720us) server_negotiation.cc:407] Connection header received
0620 07:16:40.767777 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.767782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.767958 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.768091 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:40.768332 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.769470 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.770906 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.771701 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.771840 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.771968 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.772492 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.772595 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":98,"threads_started":1}
I20260620 07:16:40.772428 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.766385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:37005 (local address 127.0.0.1:55354)
0620 07:16:40.767510 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:40.767523 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:40.767604 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:40.768091 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:40.768093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:40.768145 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:40.768221 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.768226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.769595 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.769598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.770778 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.770788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.772033 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.772036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.772100 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.772117 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 07:16:40.772206 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 07:16:40.772281 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":36,"thread_start_us":105,"threads_started":1}
I20260620 07:16:40.775243 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 9db28e831cca4b50978c3b834070948d (DEFAULT_TABLE table=test-table [id=71af4c1f7a644491970362cbb2082621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:40.775243 13011 tablet_service.cc:1467] Processing CreateTablet for tablet 9db28e831cca4b50978c3b834070948d (DEFAULT_TABLE table=test-table [id=71af4c1f7a644491970362cbb2082621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:40.775573 13151 tablet_service.cc:1467] Processing CreateTablet for tablet 9db28e831cca4b50978c3b834070948d (DEFAULT_TABLE table=test-table [id=71af4c1f7a644491970362cbb2082621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:40.776332 13011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db28e831cca4b50978c3b834070948d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:40.776335 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db28e831cca4b50978c3b834070948d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:40.776475 13151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db28e831cca4b50978c3b834070948d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:40.782884 13380 tablet_bootstrap.cc:492] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13: Bootstrap starting.
I20260620 07:16:40.784466 13381 tablet_bootstrap.cc:492] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530: Bootstrap starting.
I20260620 07:16:40.784464 13382 tablet_bootstrap.cc:492] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6: Bootstrap starting.
I20260620 07:16:40.784992 13380 tablet_bootstrap.cc:654] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:40.785781 13380 log.cc:824] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:40.786607 13381 tablet_bootstrap.cc:654] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:40.786607 13382 tablet_bootstrap.cc:654] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:40.787456 13382 log.cc:824] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:40.787695 13381 log.cc:824] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:40.787886 13380 tablet_bootstrap.cc:492] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13: No bootstrap required, opened a new log
I20260620 07:16:40.788095 13380 ts_tablet_manager.cc:1397] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:16:40.789194 13382 tablet_bootstrap.cc:492] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6: No bootstrap required, opened a new log
I20260620 07:16:40.789386 13382 ts_tablet_manager.cc:1397] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:16:40.789618 13381 tablet_bootstrap.cc:492] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530: No bootstrap required, opened a new log
I20260620 07:16:40.789800 13381 ts_tablet_manager.cc:1397] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:16:40.791998 13380 raft_consensus.cc:348] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } }
I20260620 07:16:40.792119 13382 raft_consensus.cc:348] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } }
I20260620 07:16:40.792279 13380 raft_consensus.cc:374] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:40.792328 13380 raft_consensus.cc:729] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5fdbaf614e46c0ba582fbe3d2e1d13, State: Initialized, Role: FOLLOWER
I20260620 07:16:40.792357 13382 raft_consensus.cc:374] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:40.792497 13382 raft_consensus.cc:729] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5f98910cee49b2b4d4e9591dab9fb6, State: Initialized, Role: FOLLOWER
I20260620 07:16:40.792860 13380 consensus_queue.cc:260] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } }
I20260620 07:16:40.792896 13382 consensus_queue.cc:260] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } }
I20260620 07:16:40.793474 13357 heartbeater.cc:503] Master 127.11.123.190:36289 was elected leader, sending a full tablet report...
I20260620 07:16:40.793823 13382 ts_tablet_manager.cc:1428] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260620 07:16:40.793844 13380 ts_tablet_manager.cc:1428] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260620 07:16:40.794149 13217 heartbeater.cc:503] Master 127.11.123.190:36289 was elected leader, sending a full tablet report...
I20260620 07:16:40.793509 13381 raft_consensus.cc:348] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } }
I20260620 07:16:40.794363 13381 raft_consensus.cc:374] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:40.794445 13381 raft_consensus.cc:729] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db88f9ca298d4eac82a64b81c514c530, State: Initialized, Role: FOLLOWER
I20260620 07:16:40.794854 13381 consensus_queue.cc:260] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } }
I20260620 07:16:40.795509 13077 heartbeater.cc:503] Master 127.11.123.190:36289 was elected leader, sending a full tablet report...
I20260620 07:16:40.795758 13381 ts_tablet_manager.cc:1428] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 07:16:40.803220 13078 tablet.cc:2369] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:16:40.941721 13386 raft_consensus.cc:482] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:16:40.941962 13386 raft_consensus.cc:504] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } }
I20260620 07:16:40.943166 13386 leader_election.cc:288] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db88f9ca298d4eac82a64b81c514c530 (127.11.123.129:42805), db5fdbaf614e46c0ba582fbe3d2e1d13 (127.11.123.130:37005)
I20260620 07:16:40.947124 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.943435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:37005 (local address 127.11.123.131:37965)
0620 07:16:40.943577 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:40.943593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:40.943698 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:40.944005 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:40.944011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:40.944030 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:40.944139 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.944144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.944940 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.944943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.946053 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.946063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.946806 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.946808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.946890 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.946909 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 07:16:40.946967 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 07:16:40.947024 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 07:16:40.947203 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.943688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:37965 (local address 127.11.123.130:37005)
0620 07:16:40.943810 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.943813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.943826 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:16:40.943875 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.943877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.943913 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.943985 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:40.944270 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.944831 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.946196 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.946690 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.946820 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.946870 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.947007 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.947080 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:16:40.947853 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.943447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:42805 (local address 127.11.123.131:43457)
0620 07:16:40.943811 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:40.943821 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:40.943925 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:40.944239 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:40.944244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:40.944262 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:40.944734 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.944741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.945490 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.945493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.946444 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:40.946451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.947432 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.947435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:40.947572 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.947598 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 07:16:40.947674 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 07:16:40.947730 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":54,"thread_start_us":134,"threads_started":1}
I20260620 07:16:40.947854 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:40.943507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:43457 (local address 127.11.123.129:42805)
0620 07:16:40.943630 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:40.943636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:40.943896 (+ 260us) server_negotiation.cc:407] Connection header received
0620 07:16:40.944086 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:40.944092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:40.944142 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:40.944219 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:40.944857 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.945387 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.946572 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:40.947187 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:40.947310 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:40.947368 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:40.947703 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 07:16:40.947767 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:16:40.947845 13171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db28e831cca4b50978c3b834070948d" candidate_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" is_pre_election: true
I20260620 07:16:40.948181 13171 raft_consensus.cc:2454] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de5f98910cee49b2b4d4e9591dab9fb6 in term 0.
I20260620 07:16:40.948563 13031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db28e831cca4b50978c3b834070948d" candidate_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db88f9ca298d4eac82a64b81c514c530" is_pre_election: true
I20260620 07:16:40.948689 13242 leader_election.cc:302] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db5fdbaf614e46c0ba582fbe3d2e1d13, de5f98910cee49b2b4d4e9591dab9fb6; no voters:
I20260620 07:16:40.948845 13031 raft_consensus.cc:2454] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de5f98910cee49b2b4d4e9591dab9fb6 in term 0.
I20260620 07:16:40.949056 13386 raft_consensus.cc:2781] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:16:40.949198 13386 raft_consensus.cc:482] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:16:40.949256 13386 raft_consensus.cc:3037] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:40.950307 13386 raft_consensus.cc:504] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } }
I20260620 07:16:40.950644 13386 leader_election.cc:288] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [CANDIDATE]: Term 1 election: Requested vote from peers db88f9ca298d4eac82a64b81c514c530 (127.11.123.129:42805), db5fdbaf614e46c0ba582fbe3d2e1d13 (127.11.123.130:37005)
I20260620 07:16:40.950935 13031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db28e831cca4b50978c3b834070948d" candidate_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db88f9ca298d4eac82a64b81c514c530"
I20260620 07:16:40.951009 13171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db28e831cca4b50978c3b834070948d" candidate_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13"
I20260620 07:16:40.951058 13031 raft_consensus.cc:3037] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:40.951118 13171 raft_consensus.cc:3037] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:40.952337 13171 raft_consensus.cc:2454] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de5f98910cee49b2b4d4e9591dab9fb6 in term 1.
I20260620 07:16:40.952328 13031 raft_consensus.cc:2454] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de5f98910cee49b2b4d4e9591dab9fb6 in term 1.
I20260620 07:16:40.952654 13242 leader_election.cc:302] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db5fdbaf614e46c0ba582fbe3d2e1d13, de5f98910cee49b2b4d4e9591dab9fb6; no voters:
I20260620 07:16:40.952850 13386 raft_consensus.cc:2781] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:40.953068 13386 raft_consensus.cc:686] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 1 LEADER]: Becoming Leader. State: Replica: de5f98910cee49b2b4d4e9591dab9fb6, State: Running, Role: LEADER
I20260620 07:16:40.953462 13386 consensus_queue.cc:237] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } }
I20260620 07:16:40.956426 12872 catalog_manager.cc:5495] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to de5f98910cee49b2b4d4e9591dab9fb6 (127.11.123.131). New cstate: current_term: 1 leader_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } health_report { overall_health: UNKNOWN } } }
W20260620 07:16:40.967557 13358 tablet.cc:2369] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:16:41.009262 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.005769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.11.123.131:34943)
0620 07:16:41.005893 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:41.005897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:41.005911 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:16:41.006077 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:41.006079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:41.006132 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:41.006223 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:41.006431 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.007014 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.007863 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.008129 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.008253 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.008494 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:41.008875 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:41.008980 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:41.009059 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 07:16:41.009155 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:16:41.014734 13171 raft_consensus.cc:1261] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 1 FOLLOWER]: Refusing update from remote peer de5f98910cee49b2b4d4e9591dab9fb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:41.014748 13031 raft_consensus.cc:1261] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530 [term 1 FOLLOWER]: Refusing update from remote peer de5f98910cee49b2b4d4e9591dab9fb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:41.015403 13386 consensus_queue.cc:1035] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:41.015651 13390 consensus_queue.cc:1035] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:16:41.016986 13218 tablet.cc:2369] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:16:41.022620 13393 mvcc.cc:205] Tried to move back new op lower bound from 7298825424949952512 to 7298825424708001792. Current Snapshot: MvccSnapshot[applied={T|T < 7298825424949952512}]
I20260620 07:16:41.023835 13394 mvcc.cc:205] Tried to move back new op lower bound from 7298825424949952512 to 7298825424708001792. Current Snapshot: MvccSnapshot[applied={T|T < 7298825424949952512}]
I20260620 07:16:41.024845 13395 mvcc.cc:205] Tried to move back new op lower bound from 7298825424949952512 to 7298825424708001792. Current Snapshot: MvccSnapshot[applied={T|T < 7298825424949952512}]
I20260620 07:16:41.033746 12872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:16:41.033887 12872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:16:41.036188 12872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20: Sending DeleteTablet for 3 replicas of tablet 9db28e831cca4b50978c3b834070948d
I20260620 07:16:41.037012 13289 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db28e831cca4b50978c3b834070948d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:41 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:53606
I20260620 07:16:41.037040 13151 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db28e831cca4b50978c3b834070948d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:41 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:55354
I20260620 07:16:41.037006 13011 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db28e831cca4b50978c3b834070948d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:41 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:42246
I20260620 07:16:41.037678 13405 tablet_replica.cc:330] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6: stopping tablet replica
I20260620 07:16:41.037688 13403 tablet_replica.cc:330] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13: stopping tablet replica
I20260620 07:16:41.037967 13403 raft_consensus.cc:2229] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:16:41.037969 13405 raft_consensus.cc:2229] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:16:41.038252 13403 raft_consensus.cc:2258] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:41.038419 13405 raft_consensus.cc:2258] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:41.038578 12872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:16:41.038983 12872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:16:41.039755 13403 ts_tablet_manager.cc:1905] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:41.039812 13405 ts_tablet_manager.cc:1905] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:41.041893 13404 tablet_replica.cc:330] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530: stopping tablet replica
I20260620 07:16:41.042191 13404 raft_consensus.cc:2229] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:16:41.042472 13404 raft_consensus.cc:2258] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:41.043607 13403 ts_tablet_manager.cc:1918] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:16:41.043689 13403 log.cc:1196] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/9db28e831cca4b50978c3b834070948d
I20260620 07:16:41.043871 13011 tablet_service.cc:1467] Processing CreateTablet for tablet 5ecb5c373bf5493a9bc6211dda64bab9 (DEFAULT_TABLE table=test-table [id=3dd1b61a4bf64dd58fd3d6b7d1a16d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:41.043982 13403 ts_tablet_manager.cc:1939] T 9db28e831cca4b50978c3b834070948d P db5fdbaf614e46c0ba582fbe3d2e1d13: Deleting consensus metadata
I20260620 07:16:41.044132 13011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ecb5c373bf5493a9bc6211dda64bab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:41.045826 12858 catalog_manager.cc:4841] TS db5fdbaf614e46c0ba582fbe3d2e1d13 (127.11.123.130:37005): tablet 9db28e831cca4b50978c3b834070948d (table test-table [id=71af4c1f7a644491970362cbb2082621]) successfully deleted
I20260620 07:16:41.046417 13151 tablet_service.cc:1467] Processing CreateTablet for tablet 5ecb5c373bf5493a9bc6211dda64bab9 (DEFAULT_TABLE table=test-table [id=3dd1b61a4bf64dd58fd3d6b7d1a16d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:41.046571 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 5ecb5c373bf5493a9bc6211dda64bab9 (DEFAULT_TABLE table=test-table [id=3dd1b61a4bf64dd58fd3d6b7d1a16d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:41.046660 13151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ecb5c373bf5493a9bc6211dda64bab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:41.046802 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ecb5c373bf5493a9bc6211dda64bab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:41.048763 13405 ts_tablet_manager.cc:1918] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:16:41.048826 13380 tablet_bootstrap.cc:492] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13: Bootstrap starting.
I20260620 07:16:41.048935 13405 log.cc:1196] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/9db28e831cca4b50978c3b834070948d
I20260620 07:16:41.049152 13381 tablet_bootstrap.cc:492] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530: Bootstrap starting.
I20260620 07:16:41.049391 13405 ts_tablet_manager.cc:1939] T 9db28e831cca4b50978c3b834070948d P de5f98910cee49b2b4d4e9591dab9fb6: Deleting consensus metadata
I20260620 07:16:41.049856 13380 tablet_bootstrap.cc:654] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:41.050076 13381 tablet_bootstrap.cc:654] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:41.050611 13382 tablet_bootstrap.cc:492] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6: Bootstrap starting.
I20260620 07:16:41.051458 13382 tablet_bootstrap.cc:654] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:41.051592 13404 ts_tablet_manager.cc:1905] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:41.052346 12858 catalog_manager.cc:4841] TS de5f98910cee49b2b4d4e9591dab9fb6 (127.11.123.131:34943): tablet 9db28e831cca4b50978c3b834070948d (table test-table [id=71af4c1f7a644491970362cbb2082621]) successfully deleted
I20260620 07:16:41.053323 13382 tablet_bootstrap.cc:492] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6: No bootstrap required, opened a new log
I20260620 07:16:41.053396 13382 ts_tablet_manager.cc:1397] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 07:16:41.053791 13382 raft_consensus.cc:348] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.053908 13382 raft_consensus.cc:374] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:41.053943 13382 raft_consensus.cc:729] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5f98910cee49b2b4d4e9591dab9fb6, State: Initialized, Role: FOLLOWER
I20260620 07:16:41.054042 13382 consensus_queue.cc:260] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.054304 13382 ts_tablet_manager.cc:1428] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:16:41.054800 13404 ts_tablet_manager.cc:1918] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:16:41.054944 13404 log.cc:1196] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/wal/wals/9db28e831cca4b50978c3b834070948d
I20260620 07:16:41.055281 13404 ts_tablet_manager.cc:1939] T 9db28e831cca4b50978c3b834070948d P db88f9ca298d4eac82a64b81c514c530: Deleting consensus metadata
I20260620 07:16:41.056183 13381 tablet_bootstrap.cc:492] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530: No bootstrap required, opened a new log
I20260620 07:16:41.056262 13381 ts_tablet_manager.cc:1397] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 07:16:41.056419 12859 catalog_manager.cc:4841] TS db88f9ca298d4eac82a64b81c514c530 (127.11.123.129:42805): tablet 9db28e831cca4b50978c3b834070948d (table test-table [id=71af4c1f7a644491970362cbb2082621]) successfully deleted
I20260620 07:16:41.056645 13381 raft_consensus.cc:348] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.056749 13381 raft_consensus.cc:374] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:41.056780 13381 raft_consensus.cc:729] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db88f9ca298d4eac82a64b81c514c530, State: Initialized, Role: FOLLOWER
I20260620 07:16:41.056890 13381 consensus_queue.cc:260] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.057133 13381 ts_tablet_manager.cc:1428] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:16:41.057545 13380 tablet_bootstrap.cc:492] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13: No bootstrap required, opened a new log
I20260620 07:16:41.057632 13380 ts_tablet_manager.cc:1397] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260620 07:16:41.058065 13380 raft_consensus.cc:348] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.058187 13380 raft_consensus.cc:374] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:41.058225 13380 raft_consensus.cc:729] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5fdbaf614e46c0ba582fbe3d2e1d13, State: Initialized, Role: FOLLOWER
I20260620 07:16:41.058360 13380 consensus_queue.cc:260] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.058640 13380 ts_tablet_manager.cc:1428] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:16:41.202782 13387 raft_consensus.cc:482] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:16:41.202915 13387 raft_consensus.cc:504] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.203974 13387 leader_election.cc:288] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db88f9ca298d4eac82a64b81c514c530 (127.11.123.129:42805), de5f98910cee49b2b4d4e9591dab9fb6 (127.11.123.131:34943)
I20260620 07:16:41.207679 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.204266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:44487 (local address 127.11.123.129:42805)
0620 07:16:41.204423 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:41.204427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:41.204581 (+ 154us) server_negotiation.cc:407] Connection header received
0620 07:16:41.204746 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:41.204748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:41.204790 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:41.204871 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:41.205179 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.205672 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.206711 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.207092 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.207262 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.207317 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:41.207495 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 07:16:41.207561 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:16:41.207693 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.204121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:42805 (local address 127.11.123.130:44487)
0620 07:16:41.204430 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:41.204493 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:41.204627 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:41.204882 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:41.204884 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:41.204899 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:41.205027 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:41.205033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.205777 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.205783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:41.206589 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:41.206598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.207259 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.207263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:41.207365 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.207386 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:16:41.207479 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 07:16:41.207547 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":80,"threads_started":1}
I20260620 07:16:41.207798 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.204340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:54365 (local address 127.11.123.131:34943)
0620 07:16:41.204522 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:41.204527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:41.204686 (+ 159us) server_negotiation.cc:407] Connection header received
0620 07:16:41.204837 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:41.204839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:41.204878 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:41.204953 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:41.205270 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.205745 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.206818 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.207238 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.207364 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.207414 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:41.207598 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 07:16:41.207675 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:16:41.207825 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.204232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:34943 (local address 127.11.123.130:54365)
0620 07:16:41.204586 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:41.204597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:41.204718 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:41.204982 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:41.204984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:41.204999 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:41.205161 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:41.205167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.205865 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.205870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:41.206710 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:41.206718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.207378 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.207381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:41.207478 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.207503 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:16:41.207584 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 07:16:41.207650 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":77,"threads_started":1}
I20260620 07:16:41.208288 13031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ecb5c373bf5493a9bc6211dda64bab9" candidate_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db88f9ca298d4eac82a64b81c514c530" is_pre_election: true
I20260620 07:16:41.208338 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ecb5c373bf5493a9bc6211dda64bab9" candidate_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" is_pre_election: true
I20260620 07:16:41.208443 13031 raft_consensus.cc:2454] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db5fdbaf614e46c0ba582fbe3d2e1d13 in term 0.
I20260620 07:16:41.208593 13305 raft_consensus.cc:2454] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db5fdbaf614e46c0ba582fbe3d2e1d13 in term 0.
I20260620 07:16:41.208855 13103 leader_election.cc:302] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db5fdbaf614e46c0ba582fbe3d2e1d13, db88f9ca298d4eac82a64b81c514c530; no voters:
I20260620 07:16:41.209136 13387 raft_consensus.cc:2781] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:16:41.209196 13387 raft_consensus.cc:482] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:16:41.209240 13387 raft_consensus.cc:3037] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:41.210208 13387 raft_consensus.cc:504] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.210569 13387 leader_election.cc:288] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [CANDIDATE]: Term 1 election: Requested vote from peers db88f9ca298d4eac82a64b81c514c530 (127.11.123.129:42805), de5f98910cee49b2b4d4e9591dab9fb6 (127.11.123.131:34943)
I20260620 07:16:41.210865 13031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ecb5c373bf5493a9bc6211dda64bab9" candidate_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db88f9ca298d4eac82a64b81c514c530"
I20260620 07:16:41.210894 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ecb5c373bf5493a9bc6211dda64bab9" candidate_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de5f98910cee49b2b4d4e9591dab9fb6"
I20260620 07:16:41.210965 13031 raft_consensus.cc:3037] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:41.211026 13305 raft_consensus.cc:3037] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:41.211913 13031 raft_consensus.cc:2454] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db5fdbaf614e46c0ba582fbe3d2e1d13 in term 1.
I20260620 07:16:41.212037 13305 raft_consensus.cc:2454] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db5fdbaf614e46c0ba582fbe3d2e1d13 in term 1.
I20260620 07:16:41.212173 13103 leader_election.cc:302] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db5fdbaf614e46c0ba582fbe3d2e1d13, db88f9ca298d4eac82a64b81c514c530; no voters:
I20260620 07:16:41.212323 13387 raft_consensus.cc:2781] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:41.212539 13387 raft_consensus.cc:686] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 1 LEADER]: Becoming Leader. State: Replica: db5fdbaf614e46c0ba582fbe3d2e1d13, State: Running, Role: LEADER
I20260620 07:16:41.212775 13387 consensus_queue.cc:237] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.214917 12872 catalog_manager.cc:5495] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 reported cstate change: term changed from 0 to 1, leader changed from <none> to db5fdbaf614e46c0ba582fbe3d2e1d13 (127.11.123.130). New cstate: current_term: 1 leader_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } health_report { overall_health: UNKNOWN } } }
Jun 20 07:16:41 dist-test-slave-j663 krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939799, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:41.282191 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.266701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36289 (local address 127.0.0.1:32956)
0620 07:16:41.267166 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:41.267178 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:41.267305 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:41.267576 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:41.267579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:41.267975 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:41.268135 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:41.268145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.268876 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.268879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:41.269549 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:41.269557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.269995 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.270003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:41.270068 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.277506 (+ 7438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:41.277521 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:41.278920 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:41.280534 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:41.280541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:41.280547 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:41.280768 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:41.281024 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:41.281026 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:41.281027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:41.281158 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:41.281598 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:41.281603 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:41.281701 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 07:16:41.281789 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 07:16:41.281964 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":156,"threads_started":1}
I20260620 07:16:41.282209 13418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.266755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.11.123.190:36289)
0620 07:16:41.267097 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:41.267108 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:41.267246 (+ 138us) server_negotiation.cc:407] Connection header received
0620 07:16:41.267396 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:41.267398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:41.267442 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:41.267537 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:41.268243 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.268775 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.269664 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.269877 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.269967 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.279048 (+ 9081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:41.279052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:41.279054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:41.279075 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:41.280433 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:41.280853 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:41.280855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:41.280857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:41.280937 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:41.281240 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:41.281243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:41.281245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:41.281502 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:41.281582 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:41.281855 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 07:16:41.282053 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":245,"thread_start_us":121,"threads_started":1}
I20260620 07:16:41.285784 12872 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:32956:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 07:16:41.286547 12872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:16:41.291301 13011 tablet_service.cc:1467] Processing CreateTablet for tablet a9d125d6382d4e50bad2d3dbc56a5718 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b16aa9ac6acb4beb8f749f31b032e8cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:16:41.291452 13151 tablet_service.cc:1467] Processing CreateTablet for tablet a9d125d6382d4e50bad2d3dbc56a5718 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b16aa9ac6acb4beb8f749f31b032e8cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:16:41.291491 13289 tablet_service.cc:1467] Processing CreateTablet for tablet a9d125d6382d4e50bad2d3dbc56a5718 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b16aa9ac6acb4beb8f749f31b032e8cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:16:41.291742 13011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d125d6382d4e50bad2d3dbc56a5718. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:41.291813 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d125d6382d4e50bad2d3dbc56a5718. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:41.291848 13151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d125d6382d4e50bad2d3dbc56a5718. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:41.294276 13380 tablet_bootstrap.cc:492] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: Bootstrap starting.
I20260620 07:16:41.294574 13381 tablet_bootstrap.cc:492] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530: Bootstrap starting.
I20260620 07:16:41.295186 13380 tablet_bootstrap.cc:654] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:41.295472 13382 tablet_bootstrap.cc:492] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6: Bootstrap starting.
I20260620 07:16:41.295480 13381 tablet_bootstrap.cc:654] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:41.296422 13382 tablet_bootstrap.cc:654] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:41.296532 13380 tablet_bootstrap.cc:492] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: No bootstrap required, opened a new log
I20260620 07:16:41.296613 13380 ts_tablet_manager.cc:1397] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:16:41.297068 13380 raft_consensus.cc:348] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.297240 13380 raft_consensus.cc:374] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:41.297283 13380 raft_consensus.cc:729] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5fdbaf614e46c0ba582fbe3d2e1d13, State: Initialized, Role: FOLLOWER
I20260620 07:16:41.297421 13380 consensus_queue.cc:260] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.297540 13381 tablet_bootstrap.cc:492] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530: No bootstrap required, opened a new log
I20260620 07:16:41.297547 13382 tablet_bootstrap.cc:492] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6: No bootstrap required, opened a new log
I20260620 07:16:41.297607 13381 ts_tablet_manager.cc:1397] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 07:16:41.297622 13382 ts_tablet_manager.cc:1397] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:16:41.297719 13380 ts_tablet_manager.cc:1428] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:16:41.297890 13387 tablet_replica.cc:439] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } }
I20260620 07:16:41.298007 13387 tablet_replica.cc:442] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:41.298038 13381 raft_consensus.cc:348] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.298036 13382 raft_consensus.cc:348] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.298130 13381 raft_consensus.cc:374] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:41.298147 13382 raft_consensus.cc:374] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:41.298167 13381 raft_consensus.cc:729] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db88f9ca298d4eac82a64b81c514c530, State: Initialized, Role: FOLLOWER
I20260620 07:16:41.298180 13382 raft_consensus.cc:729] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5f98910cee49b2b4d4e9591dab9fb6, State: Initialized, Role: FOLLOWER
I20260620 07:16:41.298293 13382 consensus_queue.cc:260] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.298310 13381 consensus_queue.cc:260] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.298565 13382 ts_tablet_manager.cc:1428] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:16:41.298565 13381 ts_tablet_manager.cc:1428] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:16:41.298604 13388 tablet_replica.cc:439] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } }
I20260620 07:16:41.298729 13388 tablet_replica.cc:442] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:41.298789 13398 tablet_replica.cc:439] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } }
I20260620 07:16:41.298894 13398 tablet_replica.cc:442] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:41.385771 13387 raft_consensus.cc:482] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:16:41.385917 13387 raft_consensus.cc:504] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.386441 13387 leader_election.cc:288] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db88f9ca298d4eac82a64b81c514c530 (127.11.123.129:42805), de5f98910cee49b2b4d4e9591dab9fb6 (127.11.123.131:34943)
I20260620 07:16:41.386914 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d125d6382d4e50bad2d3dbc56a5718" candidate_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" is_pre_election: true
I20260620 07:16:41.386910 13031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d125d6382d4e50bad2d3dbc56a5718" candidate_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db88f9ca298d4eac82a64b81c514c530" is_pre_election: true
I20260620 07:16:41.387071 13031 raft_consensus.cc:2454] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db5fdbaf614e46c0ba582fbe3d2e1d13 in term 0.
I20260620 07:16:41.387071 13305 raft_consensus.cc:2454] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db5fdbaf614e46c0ba582fbe3d2e1d13 in term 0.
I20260620 07:16:41.387362 13102 leader_election.cc:302] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db5fdbaf614e46c0ba582fbe3d2e1d13, de5f98910cee49b2b4d4e9591dab9fb6; no voters:
I20260620 07:16:41.387519 13387 raft_consensus.cc:2781] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:16:41.387578 13387 raft_consensus.cc:482] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:16:41.387622 13387 raft_consensus.cc:3037] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:41.388589 13387 raft_consensus.cc:504] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.388911 13387 leader_election.cc:288] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [CANDIDATE]: Term 1 election: Requested vote from peers db88f9ca298d4eac82a64b81c514c530 (127.11.123.129:42805), de5f98910cee49b2b4d4e9591dab9fb6 (127.11.123.131:34943)
I20260620 07:16:41.389195 13031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d125d6382d4e50bad2d3dbc56a5718" candidate_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db88f9ca298d4eac82a64b81c514c530"
I20260620 07:16:41.389323 13031 raft_consensus.cc:3037] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:41.389293 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d125d6382d4e50bad2d3dbc56a5718" candidate_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de5f98910cee49b2b4d4e9591dab9fb6"
I20260620 07:16:41.389420 13305 raft_consensus.cc:3037] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:41.390365 13031 raft_consensus.cc:2454] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db5fdbaf614e46c0ba582fbe3d2e1d13 in term 1.
I20260620 07:16:41.390524 13305 raft_consensus.cc:2454] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db5fdbaf614e46c0ba582fbe3d2e1d13 in term 1.
I20260620 07:16:41.390631 13103 leader_election.cc:302] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db5fdbaf614e46c0ba582fbe3d2e1d13, db88f9ca298d4eac82a64b81c514c530; no voters:
I20260620 07:16:41.390861 13387 raft_consensus.cc:2781] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:41.390961 13387 raft_consensus.cc:686] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 1 LEADER]: Becoming Leader. State: Replica: db5fdbaf614e46c0ba582fbe3d2e1d13, State: Running, Role: LEADER
I20260620 07:16:41.391091 13387 consensus_queue.cc:237] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } }
I20260620 07:16:41.391749 13411 tablet_replica.cc:439] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: TxnStatusTablet state changed. Reason: New leader db5fdbaf614e46c0ba582fbe3d2e1d13. Latest consensus state: current_term: 1 leader_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } }
I20260620 07:16:41.391852 13411 tablet_replica.cc:442] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:16:41.392194 13424 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 07:16:41.392366 13424 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 07:16:41.392765 12873 catalog_manager.cc:5495] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 reported cstate change: term changed from 0 to 1, leader changed from <none> to db5fdbaf614e46c0ba582fbe3d2e1d13 (127.11.123.130). New cstate: current_term: 1 leader_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } health_report { overall_health: UNKNOWN } } }
I20260620 07:16:41.424206 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.419924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:37005 (local address 127.0.0.1:55370)
0620 07:16:41.420265 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:41.420280 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:41.420396 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:41.420745 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:41.420748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:41.420755 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:16:41.420846 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:41.420850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.421903 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.421907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:41.422502 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:41.422510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.423082 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.423085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:41.423159 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.423203 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:16:41.423953 (+ 750us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:16:41.423965 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 07:16:41.424023 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 07:16:41.424094 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260620 07:16:41.424244 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.420081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.11.123.130:37005)
0620 07:16:41.420196 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:41.420200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:41.420358 (+ 158us) server_negotiation.cc:407] Connection header received
0620 07:16:41.420542 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:41.420546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:41.420637 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:41.420738 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:41.420951 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.421784 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.422618 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.422955 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.423062 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.423348 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:41.423851 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:41.423961 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:41.424043 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 07:16:41.424094 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 07:16:41.426964 13031 raft_consensus.cc:1261] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530 [term 1 FOLLOWER]: Refusing update from remote peer db5fdbaf614e46c0ba582fbe3d2e1d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:41.427165 13305 raft_consensus.cc:1261] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6 [term 1 FOLLOWER]: Refusing update from remote peer db5fdbaf614e46c0ba582fbe3d2e1d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:41.427440 13411 consensus_queue.cc:1035] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:41.427822 13411 consensus_queue.cc:1035] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:41.428843 13388 tablet_replica.cc:439] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530: TxnStatusTablet state changed. Reason: New leader db5fdbaf614e46c0ba582fbe3d2e1d13. Latest consensus state: current_term: 1 leader_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } }
I20260620 07:16:41.428951 13388 tablet_replica.cc:442] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:41.429718 13398 tablet_replica.cc:439] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6: TxnStatusTablet state changed. Reason: New leader db5fdbaf614e46c0ba582fbe3d2e1d13. Latest consensus state: current_term: 1 leader_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } }
I20260620 07:16:41.429829 13398 tablet_replica.cc:442] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:41.430470 13387 tablet_replica.cc:439] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } }
I20260620 07:16:41.430564 13387 tablet_replica.cc:442] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:16:41.431845 13411 tablet_replica.cc:439] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } }
I20260620 07:16:41.431936 13411 tablet_replica.cc:442] T a9d125d6382d4e50bad2d3dbc56a5718 P db5fdbaf614e46c0ba582fbe3d2e1d13: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:16:41.431986 13388 tablet_replica.cc:439] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } }
I20260620 07:16:41.432094 13388 tablet_replica.cc:442] T a9d125d6382d4e50bad2d3dbc56a5718 P db88f9ca298d4eac82a64b81c514c530: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:41.432755 13398 tablet_replica.cc:439] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 } } peers { permanent_uuid: "db5fdbaf614e46c0ba582fbe3d2e1d13" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 37005 } } peers { permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 } } }
I20260620 07:16:41.432850 13398 tablet_replica.cc:442] T a9d125d6382d4e50bad2d3dbc56a5718 P de5f98910cee49b2b4d4e9591dab9fb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:41.440236 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.437053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.11.123.130:37005)
0620 07:16:41.437163 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:41.437168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:41.437238 (+ 70us) server_negotiation.cc:407] Connection header received
0620 07:16:41.437373 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:41.437376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:41.437420 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:41.437497 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:41.437728 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.438219 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.439138 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.439454 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.439554 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.439802 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:41.439898 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:41.439974 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:41.440073 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 07:16:41.440117 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:16:41.450744 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.447014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:44349 (local address 127.11.123.130:37005)
0620 07:16:41.447115 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:41.447118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:41.447365 (+ 247us) server_negotiation.cc:407] Connection header received
0620 07:16:41.447506 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:41.447510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:41.447565 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:41.447634 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:41.447887 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.448428 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.449697 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.449913 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.450005 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.450346 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:41.450421 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:41.450488 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:41.450596 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 07:16:41.450636 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 07:16:41.450862 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.446937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:37005 (local address 127.11.123.130:44349)
0620 07:16:41.447282 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:41.447296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:41.447394 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:41.447660 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:41.447662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:41.447670 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:16:41.447784 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:41.447790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.448531 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.448535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:41.449562 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:41.449572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.450051 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.450055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:41.450143 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.450189 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:16:41.450507 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:16:41.450513 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:16:41.450578 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 07:16:41.450649 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260620 07:16:41.457063 13305 raft_consensus.cc:1261] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6 [term 1 FOLLOWER]: Refusing update from remote peer db5fdbaf614e46c0ba582fbe3d2e1d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:41.457206 13031 raft_consensus.cc:1261] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530 [term 1 FOLLOWER]: Refusing update from remote peer db5fdbaf614e46c0ba582fbe3d2e1d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:41.457556 13387 consensus_queue.cc:1035] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db88f9ca298d4eac82a64b81c514c530" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:41.457707 13432 consensus_queue.cc:1035] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de5f98910cee49b2b4d4e9591dab9fb6" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 34943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:41.496215 12874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:16:41.496424 12874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:16:41.498327 12874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76ee2a3b32344b20a13c60e056e57b20: Sending DeleteTablet for 3 replicas of tablet 5ecb5c373bf5493a9bc6211dda64bab9
I20260620 07:16:41.498958 13011 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ecb5c373bf5493a9bc6211dda64bab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:41 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:42246
I20260620 07:16:41.498960 13289 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ecb5c373bf5493a9bc6211dda64bab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:41 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:53606
I20260620 07:16:41.499146 13405 tablet_replica.cc:330] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6: stopping tablet replica
I20260620 07:16:41.499272 13405 raft_consensus.cc:2229] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:16:41.499374 13405 raft_consensus.cc:2258] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:41.499562 13404 tablet_replica.cc:330] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530: stopping tablet replica
I20260620 07:16:41.499691 13404 raft_consensus.cc:2229] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:16:41.499806 13404 raft_consensus.cc:2258] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:41.500120 13151 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ecb5c373bf5493a9bc6211dda64bab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:41 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:55354
I20260620 07:16:41.500138 13405 ts_tablet_manager.cc:1905] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:41.500272 13403 tablet_replica.cc:330] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13: stopping tablet replica
I20260620 07:16:41.500394 13403 raft_consensus.cc:2229] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:16:41.500700 13403 raft_consensus.cc:2258] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:41.502579 13405 ts_tablet_manager.cc:1918] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:16:41.502668 13405 log.cc:1196] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/5ecb5c373bf5493a9bc6211dda64bab9
I20260620 07:16:41.502899 13405 ts_tablet_manager.cc:1939] T 5ecb5c373bf5493a9bc6211dda64bab9 P de5f98910cee49b2b4d4e9591dab9fb6: Deleting consensus metadata
I20260620 07:16:41.503536 12858 catalog_manager.cc:4841] TS de5f98910cee49b2b4d4e9591dab9fb6 (127.11.123.131:34943): tablet 5ecb5c373bf5493a9bc6211dda64bab9 (table test-table [id=3dd1b61a4bf64dd58fd3d6b7d1a16d4b]) successfully deleted
I20260620 07:16:41.504287 13403 ts_tablet_manager.cc:1905] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:41.504470 13404 ts_tablet_manager.cc:1905] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:41.507083 13403 ts_tablet_manager.cc:1918] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:16:41.507159 13403 log.cc:1196] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/5ecb5c373bf5493a9bc6211dda64bab9
I20260620 07:16:41.507220 13404 ts_tablet_manager.cc:1918] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:16:41.507287 13404 log.cc:1196] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939794025788-11758-0/minicluster-data/ts-0/wal/wals/5ecb5c373bf5493a9bc6211dda64bab9
I20260620 07:16:41.507437 13403 ts_tablet_manager.cc:1939] T 5ecb5c373bf5493a9bc6211dda64bab9 P db5fdbaf614e46c0ba582fbe3d2e1d13: Deleting consensus metadata
I20260620 07:16:41.507551 13404 ts_tablet_manager.cc:1939] T 5ecb5c373bf5493a9bc6211dda64bab9 P db88f9ca298d4eac82a64b81c514c530: Deleting consensus metadata
I20260620 07:16:41.508567 12859 catalog_manager.cc:4841] TS db88f9ca298d4eac82a64b81c514c530 (127.11.123.129:42805): tablet 5ecb5c373bf5493a9bc6211dda64bab9 (table test-table [id=3dd1b61a4bf64dd58fd3d6b7d1a16d4b]) successfully deleted
I20260620 07:16:41.508591 12858 catalog_manager.cc:4841] TS db5fdbaf614e46c0ba582fbe3d2e1d13 (127.11.123.130:37005): tablet 5ecb5c373bf5493a9bc6211dda64bab9 (table test-table [id=3dd1b61a4bf64dd58fd3d6b7d1a16d4b]) successfully deleted
Jun 20 07:16:41 dist-test-slave-j663 krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939800, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.123.129@KRBTEST.COM
I20260620 07:16:41.514971 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.500339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42258 (local address 127.11.123.129:42805)
0620 07:16:41.503501 (+ 3162us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:41.503505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:41.503518 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:16:41.503559 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:41.503562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:41.503612 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:41.503706 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:41.505182 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.505926 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.507164 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.507587 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.507666 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.511097 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:41.511107 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:41.511121 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:41.511148 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:41.513141 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:41.513616 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:41.513622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:41.513624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:41.513717 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:41.513964 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:41.513967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:41.513969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:41.514268 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:41.514471 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:41.514675 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 07:16:41.514790 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3066}
W20260620 07:16:41.518869 13071 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42258
I20260620 07:16:41.529534 13418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:41.520714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.11.123.190:36289)
0620 07:16:41.520828 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:41.520834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:41.520994 (+ 160us) server_negotiation.cc:407] Connection header received
0620 07:16:41.521143 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:41.521146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:41.521198 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:41.521285 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:41.522058 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.522822 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.523974 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:41.524331 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:41.524404 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:41.526145 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:41.526149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:41.526151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:41.526177 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:41.527799 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:41.528247 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:41.528251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:41.528252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:41.528340 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:41.528610 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:41.528613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:41.528615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:41.528831 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:41.528976 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:41.529251 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 07:16:41.529378 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 07:16:41.529884 12874 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32970
I20260620 07:16:41.532226 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 12946
I20260620 07:16:41.537703 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 13086
I20260620 07:16:41.544188 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 13226
I20260620 07:16:41.551050 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 12842
2026-06-20T07:16:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2853 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:41 dist-test-slave-j663 krb5kdc[13442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:41 dist-test-slave-j663 krb5kdc[13442](info): set up 2 sockets
Jun 20 07:16:41 dist-test-slave-j663 krb5kdc[13442](info): commencing operation
krb5kdc: starting...
W20260620 07:16:42.588812 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:16:42 dist-test-slave-j663 krb5kdc[13442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:16:42.707764 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:37289
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:35429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:37289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:42.813753 13455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:42.813997 13455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:42.814034 13455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:42.817893 13455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:42.817940 13455 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 07:16:42.817957 13455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:42.817974 13455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:42.817991 13455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:42.818005 13455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:42.822964 13455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:35429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:37289
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:37289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:42.823853 13455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:42.824749 13455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:42.832770 13461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:42.832800 13463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:42.832772 13460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:42.833113 13455 server_base.cc:1034] running on GCE node
I20260620 07:16:42.833585 13455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:42.834522 13455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:42.835671 13455 hybrid_clock.cc:648] HybridClock initialized: now 1781939802835645 us; error 40 us; skew 500 ppm
Jun 20 07:16:42 dist-test-slave-j663 krb5kdc[13442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:42.838277 13455 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:16:42.839143 13455 webserver.cc:460] Webserver started at http://127.11.123.190:38391/ using document root <none> and password file <none>
I20260620 07:16:42.839697 13455 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:42.839780 13455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:42.839995 13455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:42.841998 13455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "9934c91f29c54d698951790fb7e21511"
format_stamp: "Formatted at 2026-06-20 07:16:42 on dist-test-slave-j663"
server_key: "c4e2fb85a7431265d86aba3e376c57c5"
server_key_iv: "ecd0f7db0a62e88caa67d463f2b5b77c"
server_key_version: "encryptionkey@0"
I20260620 07:16:42.842525 13455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "9934c91f29c54d698951790fb7e21511"
format_stamp: "Formatted at 2026-06-20 07:16:42 on dist-test-slave-j663"
server_key: "c4e2fb85a7431265d86aba3e376c57c5"
server_key_iv: "ecd0f7db0a62e88caa67d463f2b5b77c"
server_key_version: "encryptionkey@0"
I20260620 07:16:42.846318 13455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:16:42.848587 13470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:42.849795 13455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:16:42.849944 13455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "9934c91f29c54d698951790fb7e21511"
format_stamp: "Formatted at 2026-06-20 07:16:42 on dist-test-slave-j663"
server_key: "c4e2fb85a7431265d86aba3e376c57c5"
server_key_iv: "ecd0f7db0a62e88caa67d463f2b5b77c"
server_key_version: "encryptionkey@0"
I20260620 07:16:42.850054 13455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:42.865484 13455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:42.868129 13455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:42.868319 13455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:42.875872 13455 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:37289
I20260620 07:16:42.875903 13532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:37289 every 8 connection(s)
I20260620 07:16:42.876757 13455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:42.879307 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:42.882745 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 13455
I20260620 07:16:42.882828 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:16:42.883075 11758 external_mini_cluster.cc:1428] Setting key eec8d1af8d69384ff24090141d467def
I20260620 07:16:42.884912 13533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511: Bootstrap starting.
I20260620 07:16:42.887032 13533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:42.887725 13533 log.cc:824] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:42.889489 13533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511: No bootstrap required, opened a new log
Jun 20 07:16:42 dist-test-slave-j663 krb5kdc[13442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:42.893297 13533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9934c91f29c54d698951790fb7e21511" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37289 } }
I20260620 07:16:42.893497 13533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:42.893580 13533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9934c91f29c54d698951790fb7e21511, State: Initialized, Role: FOLLOWER
I20260620 07:16:42.893989 13533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9934c91f29c54d698951790fb7e21511" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37289 } }
I20260620 07:16:42.894151 13533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:42.894225 13533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:42.894340 13533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:42.895143 13533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9934c91f29c54d698951790fb7e21511" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37289 } }
I20260620 07:16:42.895493 13533 leader_election.cc:302] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9934c91f29c54d698951790fb7e21511; no voters:
I20260620 07:16:42.895851 13533 leader_election.cc:288] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:42.896108 13538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:42.896348 13538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [term 1 LEADER]: Becoming Leader. State: Replica: 9934c91f29c54d698951790fb7e21511, State: Running, Role: LEADER
I20260620 07:16:42.896695 13538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9934c91f29c54d698951790fb7e21511" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37289 } }
I20260620 07:16:42.897011 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:42.884426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.11.123.190:37289)
0620 07:16:42.884826 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:42.884840 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:42.884873 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:16:42.885548 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:42.885565 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:42.885881 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:42.886204 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:42.886941 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:42.888006 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:42.888826 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:42.889333 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:42.889506 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:42.892092 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:42.892096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:42.892108 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:42.892132 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:42.893981 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:42.894425 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:42.894432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:42.894438 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:42.894563 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:42.894838 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:42.894843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:42.894845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:42.895215 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:42.895410 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:42.895713 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 07:16:42.896024 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260620 07:16:42.897044 13533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:42.898227 13539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9934c91f29c54d698951790fb7e21511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9934c91f29c54d698951790fb7e21511" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37289 } } }
I20260620 07:16:42.898340 13539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:42.898623 13540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9934c91f29c54d698951790fb7e21511. Latest consensus state: current_term: 1 leader_uuid: "9934c91f29c54d698951790fb7e21511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9934c91f29c54d698951790fb7e21511" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37289 } } }
I20260620 07:16:42.898707 13540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:42.899636 13547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:42.902139 13547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:42.907825 13547 catalog_manager.cc:1347] Generated new cluster ID: 2395414335174e50a312daa10baa1592
I20260620 07:16:42.907886 13547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:42.922031 13547 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:42.924036 13547 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:42.931669 13547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511: Generated new TSK 0
I20260620 07:16:42.932461 13547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:16:42.965844 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:37289
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:35429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:43.073426 13559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:43.073698 13559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:43.073776 13559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:43.077706 13559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:43.077773 13559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:43.077844 13559 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 07:16:43.077884 13559 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:16:43.077942 13559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:16:43.082967 13559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:35429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.123.190:37289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:43.083945 13559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:43.084894 13559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:43.094079 13565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:43.094071 13564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:43.094084 13567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:43.094376 13559 server_base.cc:1034] running on GCE node
I20260620 07:16:43.094767 13559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:43.095369 13559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:43.096555 13559 hybrid_clock.cc:648] HybridClock initialized: now 1781939803096541 us; error 31 us; skew 500 ppm
Jun 20 07:16:43 dist-test-slave-j663 krb5kdc[13442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939803, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:43.099437 13559 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:16:43.100221 13559 webserver.cc:460] Webserver started at http://127.11.123.129:46035/ using document root <none> and password file <none>
I20260620 07:16:43.100759 13559 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:43.100843 13559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:43.101013 13559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:43.103017 13559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "974f9cbe41d64215837c494a7b9d7073"
format_stamp: "Formatted at 2026-06-20 07:16:43 on dist-test-slave-j663"
server_key: "4821723e626260a1bf966014bce1eed6"
server_key_iv: "0880b7d7b310ce56ecd7ede74336a8b5"
server_key_version: "encryptionkey@0"
I20260620 07:16:43.103497 13559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "974f9cbe41d64215837c494a7b9d7073"
format_stamp: "Formatted at 2026-06-20 07:16:43 on dist-test-slave-j663"
server_key: "4821723e626260a1bf966014bce1eed6"
server_key_iv: "0880b7d7b310ce56ecd7ede74336a8b5"
server_key_version: "encryptionkey@0"
I20260620 07:16:43.107036 13559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:16:43.109251 13574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:43.110428 13559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:43.110550 13559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "974f9cbe41d64215837c494a7b9d7073"
format_stamp: "Formatted at 2026-06-20 07:16:43 on dist-test-slave-j663"
server_key: "4821723e626260a1bf966014bce1eed6"
server_key_iv: "0880b7d7b310ce56ecd7ede74336a8b5"
server_key_version: "encryptionkey@0"
I20260620 07:16:43.110630 13559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:43.125375 13559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:43.128432 13559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:43.128585 13559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:43.129933 13559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:43.129983 13559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:43.130020 13559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:43.130076 13559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:16:43 dist-test-slave-j663 krb5kdc[13442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939803, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:43.141011 13559 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:42525
I20260620 07:16:43.141037 13689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:42525 every 8 connection(s)
I20260620 07:16:43.142150 13559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:16:43.143564 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.130928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:34055 (local address 127.11.123.190:37289)
0620 07:16:43.131055 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.131059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.132064 (+ 1005us) server_negotiation.cc:407] Connection header received
0620 07:16:43.133231 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.133235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.133294 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.133426 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:43.134595 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.135121 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.136241 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.136565 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.136675 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.139540 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:43.139547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:43.139553 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:43.139587 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:43.141403 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.141859 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.141862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.141864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.141954 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.142326 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.142329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.142331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.142574 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:43.142671 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.143231 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.143380 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:16:43.144289 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.131409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37289 (local address 127.11.123.129:34055)
0620 07:16:43.131910 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:43.131952 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:43.132989 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:43.133595 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:43.133601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:43.133930 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:43.134452 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.134466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.135243 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.135247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.136116 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.136125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.136716 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.136720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.136936 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.137632 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:43.137656 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:43.139346 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:43.141509 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.141514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.141524 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.141762 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.142044 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.142046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.142048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.142169 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.142723 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:43.142730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:43.143018 (+ 288us) client_negotiation.cc:769] Sending connection context
0620 07:16:43.143171 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 07:16:43.143488 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":136,"threads_started":1}
I20260620 07:16:43.145632 13690 heartbeater.cc:347] Connected to a master server at 127.11.123.190:37289
I20260620 07:16:43.145773 13690 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:43.146226 13690 heartbeater.cc:511] Master 127.11.123.190:37289 requested a full tablet report, sending...
I20260620 07:16:43.147774 13486 ts_manager.cc:194] Registered new tserver with Master: 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129:42525)
I20260620 07:16:43.149173 13486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:34055
I20260620 07:16:43.151674 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 13559
I20260620 07:16:43.151798 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:16:43.152107 11758 external_mini_cluster.cc:1428] Setting key 620b581448484a8b95bc4a3e96cbc4fc
I20260620 07:16:43.154878 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.147700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:55871 (local address 127.11.123.190:37289)
0620 07:16:43.147819 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.147825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.148000 (+ 175us) server_negotiation.cc:407] Connection header received
0620 07:16:43.148116 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.148118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.148152 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.148214 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:43.148676 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.149250 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.150219 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.150445 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.150526 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.151674 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:43.151678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:43.151681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:43.151708 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:43.153314 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.153777 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.153780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.153781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.153850 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.154158 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.154160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.154162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.154310 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:43.154384 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.154683 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.154773 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:16:43.155097 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.147644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37289 (local address 127.11.123.129:55871)
0620 07:16:43.147916 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:43.147931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:43.148028 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:43.148246 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:43.148249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:43.148458 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:43.148576 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.148583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.149400 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.149408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.150113 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.150123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.150552 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.150556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.150639 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.151080 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:43.151093 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:43.151485 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:43.153426 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.153432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.153435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.153684 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.153964 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.153969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.153971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.154073 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.154402 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:43.154408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:43.154518 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 07:16:43.154632 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 07:16:43.154791 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:16:43.178100 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:37289
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:35429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:43.284735 13699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:43.284965 13699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:43.285008 13699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:43.288784 13699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:43.288831 13699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:43.288868 13699 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 07:16:43.288890 13699 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:16:43.288921 13699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:16:43.293789 13699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:35429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.123.190:37289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:43.294661 13699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:43.295538 13699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:43.304914 13705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:43.304932 13704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:43.304912 13707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:43.305263 13699 server_base.cc:1034] running on GCE node
I20260620 07:16:43.305698 13699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:43.306299 13699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:43.307498 13699 hybrid_clock.cc:648] HybridClock initialized: now 1781939803307473 us; error 40 us; skew 500 ppm
Jun 20 07:16:43 dist-test-slave-j663 krb5kdc[13442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939803, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:43.310214 13699 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:16:43.311091 13699 webserver.cc:460] Webserver started at http://127.11.123.130:39793/ using document root <none> and password file <none>
I20260620 07:16:43.311625 13699 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:43.311707 13699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:43.311919 13699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:43.313999 13699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "340bbb5454bc4c09a445a2b88ba886bb"
format_stamp: "Formatted at 2026-06-20 07:16:43 on dist-test-slave-j663"
server_key: "b6f486e1fe2595fec2b050f50aac88e2"
server_key_iv: "71d0d2f8b35af140d628b79b5eab835d"
server_key_version: "encryptionkey@0"
I20260620 07:16:43.314587 13699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "340bbb5454bc4c09a445a2b88ba886bb"
format_stamp: "Formatted at 2026-06-20 07:16:43 on dist-test-slave-j663"
server_key: "b6f486e1fe2595fec2b050f50aac88e2"
server_key_iv: "71d0d2f8b35af140d628b79b5eab835d"
server_key_version: "encryptionkey@0"
I20260620 07:16:43.318142 13699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:16:43.320384 13714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:43.321702 13699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:16:43.321838 13699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "340bbb5454bc4c09a445a2b88ba886bb"
format_stamp: "Formatted at 2026-06-20 07:16:43 on dist-test-slave-j663"
server_key: "b6f486e1fe2595fec2b050f50aac88e2"
server_key_iv: "71d0d2f8b35af140d628b79b5eab835d"
server_key_version: "encryptionkey@0"
I20260620 07:16:43.321946 13699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:43.349947 13699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:43.353103 13699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:43.353331 13699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:43.355072 13699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:43.355135 13699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:43.355192 13699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:43.355222 13699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:16:43 dist-test-slave-j663 krb5kdc[13442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939803, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:43.365751 13699 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:45059
I20260620 07:16:43.365828 13829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:45059 every 8 connection(s)
I20260620 07:16:43.366582 13699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:16:43.368394 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.355675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:49039 (local address 127.11.123.190:37289)
0620 07:16:43.355819 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.355825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.357022 (+ 1197us) server_negotiation.cc:407] Connection header received
0620 07:16:43.358086 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.358090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.358143 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.358224 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:43.359438 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.360040 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.361366 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.361669 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.361787 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.364411 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:43.364415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:43.364418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:43.364444 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:43.366063 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.366845 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.366850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.366852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.366948 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.367365 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.367368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.367370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.367569 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:43.367663 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.368115 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.368214 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:16:43.369275 13723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.356437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37289 (local address 127.11.123.130:49039)
0620 07:16:43.356865 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:43.356912 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:43.357883 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:43.358376 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:43.358383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:43.358689 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:43.359304 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.359319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.360156 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.360159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.361221 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.361233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.361828 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.361831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.362046 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.362615 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:43.362644 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:43.364254 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:43.366302 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.366310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.366328 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.366726 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.367090 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.367093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.367095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.367270 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.367676 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:43.367681 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:43.367910 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 07:16:43.368070 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 07:16:43.368320 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":126,"threads_started":1}
I20260620 07:16:43.370424 13830 heartbeater.cc:347] Connected to a master server at 127.11.123.190:37289
I20260620 07:16:43.370575 13830 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:43.371016 13830 heartbeater.cc:511] Master 127.11.123.190:37289 requested a full tablet report, sending...
I20260620 07:16:43.371999 13487 ts_manager.cc:194] Registered new tserver with Master: 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130:45059)
I20260620 07:16:43.372622 13487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:49039
I20260620 07:16:43.373981 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 13699
I20260620 07:16:43.374101 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:16:43.374409 11758 external_mini_cluster.cc:1428] Setting key 9cdeaccbd40fbfd4e89a7adf2086a2c8
I20260620 07:16:43.380173 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.372695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:53583 (local address 127.11.123.190:37289)
0620 07:16:43.372799 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.372803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.372992 (+ 189us) server_negotiation.cc:407] Connection header received
0620 07:16:43.373151 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.373154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.373199 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.373282 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:43.373720 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.374561 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.375716 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.376014 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.376113 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.377166 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:43.377172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:43.377178 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:43.377209 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:43.378572 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.378984 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.378989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.378991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.379072 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.379335 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.379339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.379341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.379559 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:43.379679 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.379926 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.380027 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:16:43.380185 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.372628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37289 (local address 127.11.123.130:53583)
0620 07:16:43.372898 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:43.372913 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:43.373031 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:43.373304 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:43.373307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:43.373510 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:43.373605 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.373610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.374738 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.374744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.375569 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.375579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.376135 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.376138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.376205 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.376705 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:43.376715 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:43.377013 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:43.378670 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.378672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.378673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.378884 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.379167 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.379170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.379172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.379252 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.379664 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:43.379667 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:43.379754 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 07:16:43.379863 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 07:16:43.379994 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:16:43.400131 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:37289
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:35429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:43.507567 13839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:43.507807 13839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:43.507851 13839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:43.511771 13839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:43.511819 13839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:43.511857 13839 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 07:16:43.511871 13839 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:16:43.511901 13839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:16:43.517030 13839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:35429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.123.190:37289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:43.517984 13839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:43.518896 13839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:43.528227 13847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:43.528228 13844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:43.528234 13845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:43.528332 13839 server_base.cc:1034] running on GCE node
I20260620 07:16:43.528874 13839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:43.529565 13839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:43.530742 13839 hybrid_clock.cc:648] HybridClock initialized: now 1781939803530726 us; error 27 us; skew 500 ppm
Jun 20 07:16:43 dist-test-slave-j663 krb5kdc[13442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939803, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:43.533416 13839 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:16:43.534209 13839 webserver.cc:460] Webserver started at http://127.11.123.131:36017/ using document root <none> and password file <none>
I20260620 07:16:43.534798 13839 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:43.534880 13839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:43.535081 13839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:43.537005 13839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "539f656f10e94f1080eec0cc58b15f09"
format_stamp: "Formatted at 2026-06-20 07:16:43 on dist-test-slave-j663"
server_key: "f3dd85bc488e04126f4c85d4c98876d4"
server_key_iv: "2158f06744a14a5292ad53c649705a29"
server_key_version: "encryptionkey@0"
I20260620 07:16:43.537592 13839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "539f656f10e94f1080eec0cc58b15f09"
format_stamp: "Formatted at 2026-06-20 07:16:43 on dist-test-slave-j663"
server_key: "f3dd85bc488e04126f4c85d4c98876d4"
server_key_iv: "2158f06744a14a5292ad53c649705a29"
server_key_version: "encryptionkey@0"
I20260620 07:16:43.541018 13839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:16:43.543260 13854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:43.544484 13839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:16:43.544610 13839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "539f656f10e94f1080eec0cc58b15f09"
format_stamp: "Formatted at 2026-06-20 07:16:43 on dist-test-slave-j663"
server_key: "f3dd85bc488e04126f4c85d4c98876d4"
server_key_iv: "2158f06744a14a5292ad53c649705a29"
server_key_version: "encryptionkey@0"
I20260620 07:16:43.544709 13839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:43.555912 13839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:43.558902 13839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:43.559089 13839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:43.560369 13839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:43.560442 13839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:43.560506 13839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:43.560545 13839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:16:43 dist-test-slave-j663 krb5kdc[13442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939803, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:43.570981 13839 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:33015
I20260620 07:16:43.571056 13969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:33015 every 8 connection(s)
I20260620 07:16:43.572026 13839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:16:43.574287 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.561181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:54569 (local address 127.11.123.190:37289)
0620 07:16:43.561318 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.561324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.562161 (+ 837us) server_negotiation.cc:407] Connection header received
0620 07:16:43.563144 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.563148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.563210 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.563272 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:43.564927 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.565634 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.567462 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.567822 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.567914 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.570786 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:43.570790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:43.570793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:43.570821 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:43.572389 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.572994 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.572998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.572999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.573135 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.573424 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.573427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.573428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.573602 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:43.573678 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.574079 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.574172 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:16:43.575071 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.561581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37289 (local address 127.11.123.131:54569)
0620 07:16:43.562000 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:43.562047 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:43.562904 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:43.563706 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:43.563714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:43.564087 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:43.564779 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.564801 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.565762 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.565769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.567305 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.567329 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.567955 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.567959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.568158 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.568743 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:43.568767 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:43.570558 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:43.572554 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.572559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.572573 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.572900 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.573222 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.573224 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.573225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.573347 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.573685 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:43.573690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:43.573889 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 07:16:43.574036 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 07:16:43.574295 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":120,"threads_started":1}
I20260620 07:16:43.575376 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 13839
I20260620 07:16:43.575497 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:16:43.575758 11758 external_mini_cluster.cc:1428] Setting key d9f7af9662a42e384566affee3a25cfe
I20260620 07:16:43.576349 13970 heartbeater.cc:347] Connected to a master server at 127.11.123.190:37289
I20260620 07:16:43.576501 13970 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:43.577066 13970 heartbeater.cc:511] Master 127.11.123.190:37289 requested a full tablet report, sending...
I20260620 07:16:43.578212 13487 ts_manager.cc:194] Registered new tserver with Master: 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015)
I20260620 07:16:43.578999 13487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:54569
I20260620 07:16:43.584849 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.578493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:53041 (local address 127.11.123.190:37289)
0620 07:16:43.578592 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.578596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.578986 (+ 390us) server_negotiation.cc:407] Connection header received
0620 07:16:43.579048 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.579051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.579101 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.579173 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:43.579631 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.580124 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.580922 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.581216 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.581320 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.582205 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:43.582209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:43.582211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:43.582233 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:43.583353 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.583747 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.583752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.583753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.583840 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.584098 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.584103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.584105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.584263 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:43.584363 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.584603 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.584707 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:16:43.584857 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.578685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37289 (local address 127.11.123.131:53041)
0620 07:16:43.578904 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:43.578916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:43.578993 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:43.579213 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:43.579217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:43.579422 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:43.579515 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.579521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.580242 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.580245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.580805 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.580812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.581327 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.581331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.581430 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.581801 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:43.581810 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:43.582078 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:43.583456 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.583461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.583463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.583646 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.583926 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.583930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.583933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.584012 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.584358 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:43.584361 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:43.584451 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 07:16:43.584551 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 07:16:43.584682 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260620 07:16:43.590183 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:16:43 dist-test-slave-j663 krb5kdc[13442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:16:43 dist-test-slave-j663 krb5kdc[13442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:43.613858 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.606534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.11.123.190:37289)
0620 07:16:43.606646 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.606650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.606660 (+ 10us) server_negotiation.cc:407] Connection header received
0620 07:16:43.606694 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.606696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.606733 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.606792 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:43.607432 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.607937 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.608708 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.608907 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.608978 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.611152 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:43.611155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:43.611157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:43.611178 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:43.612313 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.612671 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.612674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.612675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.612744 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.613015 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.613017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.613018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.613230 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:43.613331 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.613586 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.613669 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:16:43.616549 13487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:16:43.619654 13487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:16:43.633977 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.628251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:33015 (local address 127.0.0.1:38756)
0620 07:16:43.628676 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:43.628720 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:43.628836 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:43.629668 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:43.629675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:43.629705 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:43.629853 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.629860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.631068 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.631074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.632248 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.632258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.633432 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.633436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.633548 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.633588 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 07:16:43.633706 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 07:16:43.633830 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":58,"thread_start_us":73,"threads_started":1}
I20260620 07:16:43.634014 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.628429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.11.123.131:33015)
0620 07:16:43.629161 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.629174 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.629196 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:16:43.629270 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.629277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.629476 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.630001 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:43.630136 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.630957 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.632379 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.633292 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.633439 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.633568 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.633748 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.633841 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":111,"threads_started":1}
I20260620 07:16:43.634328 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.628303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:45059 (local address 127.0.0.1:33718)
0620 07:16:43.628911 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:43.628925 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:43.628993 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:43.630000 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:43.630004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:43.630023 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:43.630143 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.630150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.631670 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.631673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.632800 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.632808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.633954 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.633957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.634050 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.634080 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:16:43.634163 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:16:43.634220 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":36,"thread_start_us":98,"threads_started":1}
I20260620 07:16:43.634568 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.628973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:42525 (local address 127.0.0.1:56556)
0620 07:16:43.629308 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:43.629323 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:43.629423 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:43.630155 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:43.630158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:43.630176 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:43.630275 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.630282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.631839 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.631845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.632943 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.632954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.634166 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.634171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.634257 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.634276 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 07:16:43.634322 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 07:16:43.634409 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260620 07:16:43.634874 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.628918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.11.123.130:45059)
0620 07:16:43.629366 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.629380 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.629415 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:16:43.629498 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.629507 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.629701 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.629861 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:43.630274 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.631549 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.632932 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.633667 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.633786 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.633896 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.634662 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.634747 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":111,"threads_started":1}
I20260620 07:16:43.635346 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.629324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.11.123.129:42525)
0620 07:16:43.629682 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.629697 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.629719 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:16:43.629794 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.629802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.630000 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.630422 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:43.630560 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.631730 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.633123 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.633943 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.634869 (+ 926us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.635042 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.635100 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.635203 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":62,"threads_started":1}
I20260620 07:16:43.637364 13904 tablet_service.cc:1467] Processing CreateTablet for tablet 135f9dd7f3d940c6ac5d358cb274ba7a (DEFAULT_TABLE table=test-table [id=a7e842675f3841c2bf96f48bfc5d26db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:43.637612 13624 tablet_service.cc:1467] Processing CreateTablet for tablet 135f9dd7f3d940c6ac5d358cb274ba7a (DEFAULT_TABLE table=test-table [id=a7e842675f3841c2bf96f48bfc5d26db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:43.638474 13624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135f9dd7f3d940c6ac5d358cb274ba7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:43.638525 13904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135f9dd7f3d940c6ac5d358cb274ba7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:43.639011 13764 tablet_service.cc:1467] Processing CreateTablet for tablet 135f9dd7f3d940c6ac5d358cb274ba7a (DEFAULT_TABLE table=test-table [id=a7e842675f3841c2bf96f48bfc5d26db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:43.639956 13764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135f9dd7f3d940c6ac5d358cb274ba7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:43.645418 13993 tablet_bootstrap.cc:492] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073: Bootstrap starting.
I20260620 07:16:43.646641 13994 tablet_bootstrap.cc:492] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09: Bootstrap starting.
I20260620 07:16:43.647662 13993 tablet_bootstrap.cc:654] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:43.647993 13995 tablet_bootstrap.cc:492] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb: Bootstrap starting.
I20260620 07:16:43.648635 13993 log.cc:824] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:43.648708 13994 tablet_bootstrap.cc:654] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:43.649585 13994 log.cc:824] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:43.650285 13995 tablet_bootstrap.cc:654] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:43.650947 13993 tablet_bootstrap.cc:492] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073: No bootstrap required, opened a new log
I20260620 07:16:43.651149 13993 ts_tablet_manager.cc:1397] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 07:16:43.651113 13995 log.cc:824] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:43.651644 13994 tablet_bootstrap.cc:492] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09: No bootstrap required, opened a new log
I20260620 07:16:43.651844 13994 ts_tablet_manager.cc:1397] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:16:43.653102 13995 tablet_bootstrap.cc:492] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb: No bootstrap required, opened a new log
I20260620 07:16:43.653275 13995 ts_tablet_manager.cc:1397] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:16:43.654404 13993 raft_consensus.cc:348] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } }
I20260620 07:16:43.654668 13993 raft_consensus.cc:374] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:43.654719 13993 raft_consensus.cc:729] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974f9cbe41d64215837c494a7b9d7073, State: Initialized, Role: FOLLOWER
I20260620 07:16:43.655141 13993 consensus_queue.cc:260] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } }
I20260620 07:16:43.655731 13690 heartbeater.cc:503] Master 127.11.123.190:37289 was elected leader, sending a full tablet report...
I20260620 07:16:43.656077 13993 ts_tablet_manager.cc:1428] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:16:43.656003 13994 raft_consensus.cc:348] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } }
I20260620 07:16:43.656244 13994 raft_consensus.cc:374] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:43.656297 13994 raft_consensus.cc:729] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539f656f10e94f1080eec0cc58b15f09, State: Initialized, Role: FOLLOWER
I20260620 07:16:43.656716 13994 consensus_queue.cc:260] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } }
I20260620 07:16:43.657699 13994 ts_tablet_manager.cc:1428] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 07:16:43.657606 13995 raft_consensus.cc:348] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } }
I20260620 07:16:43.657853 13995 raft_consensus.cc:374] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:43.657930 13995 raft_consensus.cc:729] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 340bbb5454bc4c09a445a2b88ba886bb, State: Initialized, Role: FOLLOWER
I20260620 07:16:43.658031 13970 heartbeater.cc:503] Master 127.11.123.190:37289 was elected leader, sending a full tablet report...
I20260620 07:16:43.658373 13995 consensus_queue.cc:260] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } }
I20260620 07:16:43.658950 13830 heartbeater.cc:503] Master 127.11.123.190:37289 was elected leader, sending a full tablet report...
I20260620 07:16:43.659333 13995 ts_tablet_manager.cc:1428] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
W20260620 07:16:43.823028 13971 tablet.cc:2369] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:16:43.838840 14000 raft_consensus.cc:482] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:16:43.839025 14000 raft_consensus.cc:504] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } }
I20260620 07:16:43.840186 14000 leader_election.cc:288] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130:45059), 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129:42525)
I20260620 07:16:43.843753 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.840611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:55333 (local address 127.11.123.130:45059)
0620 07:16:43.840741 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.840747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.840763 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:16:43.840813 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.840815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.840851 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.840930 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:43.841268 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.841771 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.842811 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.843200 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.843323 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.843381 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.843574 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.843638 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:16:43.843755 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.840403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:45059 (local address 127.11.123.131:55333)
0620 07:16:43.840551 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:43.840568 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:43.840664 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:43.840953 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:43.840956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:43.841001 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:43.841158 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.841163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.841875 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.841882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.842696 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.842704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.843322 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.843325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.843445 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.843462 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 07:16:43.843547 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:16:43.843619 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":35,"spinlock_wait_cycles":41088}
I20260620 07:16:43.844355 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.840503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:42525 (local address 127.11.123.131:50095)
0620 07:16:43.840859 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:43.840869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:43.841013 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:43.841275 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:43.841277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:43.841292 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:43.841765 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.841771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.842474 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.842477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.843225 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.843232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.843979 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.843984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.844067 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.844084 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 07:16:43.844165 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 07:16:43.844225 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":20,"spinlock_wait_cycles":13568,"thread_start_us":140,"threads_started":1}
I20260620 07:16:43.844364 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.840586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:50095 (local address 127.11.123.129:42525)
0620 07:16:43.840721 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.840726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.840949 (+ 223us) server_negotiation.cc:407] Connection header received
0620 07:16:43.841137 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.841140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.841187 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.841267 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:43.841882 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.842360 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.843351 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.843736 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.843883 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.843943 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.844210 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.844267 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:16:43.844413 13784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f9dd7f3d940c6ac5d358cb274ba7a" candidate_uuid: "539f656f10e94f1080eec0cc58b15f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340bbb5454bc4c09a445a2b88ba886bb" is_pre_election: true
I20260620 07:16:43.844683 13784 raft_consensus.cc:2454] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 539f656f10e94f1080eec0cc58b15f09 in term 0.
I20260620 07:16:43.845065 13644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f9dd7f3d940c6ac5d358cb274ba7a" candidate_uuid: "539f656f10e94f1080eec0cc58b15f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974f9cbe41d64215837c494a7b9d7073" is_pre_election: true
I20260620 07:16:43.845257 13856 leader_election.cc:302] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340bbb5454bc4c09a445a2b88ba886bb, 539f656f10e94f1080eec0cc58b15f09; no voters:
I20260620 07:16:43.845372 13644 raft_consensus.cc:2454] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 539f656f10e94f1080eec0cc58b15f09 in term 0.
I20260620 07:16:43.845561 14000 raft_consensus.cc:2781] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:16:43.845654 14000 raft_consensus.cc:482] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:16:43.845709 14000 raft_consensus.cc:3037] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:43.846822 14000 raft_consensus.cc:504] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } }
I20260620 07:16:43.847157 14000 leader_election.cc:288] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [CANDIDATE]: Term 1 election: Requested vote from peers 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130:45059), 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129:42525)
I20260620 07:16:43.847465 13784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f9dd7f3d940c6ac5d358cb274ba7a" candidate_uuid: "539f656f10e94f1080eec0cc58b15f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340bbb5454bc4c09a445a2b88ba886bb"
I20260620 07:16:43.847604 13784 raft_consensus.cc:3037] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:43.848534 13784 raft_consensus.cc:2454] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 539f656f10e94f1080eec0cc58b15f09 in term 1.
I20260620 07:16:43.848798 13644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f9dd7f3d940c6ac5d358cb274ba7a" candidate_uuid: "539f656f10e94f1080eec0cc58b15f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974f9cbe41d64215837c494a7b9d7073"
I20260620 07:16:43.848853 13856 leader_election.cc:302] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340bbb5454bc4c09a445a2b88ba886bb, 539f656f10e94f1080eec0cc58b15f09; no voters:
I20260620 07:16:43.848893 13644 raft_consensus.cc:3037] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:43.849054 14000 raft_consensus.cc:2781] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:43.849324 14000 raft_consensus.cc:686] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 1 LEADER]: Becoming Leader. State: Replica: 539f656f10e94f1080eec0cc58b15f09, State: Running, Role: LEADER
I20260620 07:16:43.849653 14000 consensus_queue.cc:237] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } }
I20260620 07:16:43.850040 13644 raft_consensus.cc:2454] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 539f656f10e94f1080eec0cc58b15f09 in term 1.
I20260620 07:16:43.852686 13486 catalog_manager.cc:5495] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131). New cstate: current_term: 1 leader_uuid: "539f656f10e94f1080eec0cc58b15f09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } health_report { overall_health: UNKNOWN } } }
W20260620 07:16:43.867755 13831 tablet.cc:2369] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:16:43 dist-test-slave-j663 krb5kdc[13442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:43.890506 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.882512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37289 (local address 127.0.0.1:36360)
0620 07:16:43.882758 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:43.882771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:43.882890 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:43.883132 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:43.883135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:43.883523 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:43.883609 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.883615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.884328 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.884334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.884877 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:43.884884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.885356 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.885360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:43.885503 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.886057 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:43.886075 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:43.887565 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:43.888971 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.888981 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.888987 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.889267 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.889528 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:43.889533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:43.889534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:43.889613 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:43.889959 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:43.889965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:43.890066 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 07:16:43.890175 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 07:16:43.890319 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":39,"thread_start_us":81,"threads_started":1}
I20260620 07:16:43.890506 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:43.882573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.11.123.190:37289)
0620 07:16:43.882688 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:43.882694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:43.882843 (+ 149us) server_negotiation.cc:407] Connection header received
0620 07:16:43.882998 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:43.883002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:43.883040 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:43.883118 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:43.883723 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.884227 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.884997 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:43.885256 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:43.885360 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:43.887690 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:43.887693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:43.887696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:43.887717 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:43.888861 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.889357 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.889363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.889365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.889444 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:43.889699 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:43.889702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:43.889703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:43.889862 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:43.889961 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:43.890228 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 07:16:43.890368 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":41}
W20260620 07:16:43.893590 13691 tablet.cc:2369] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:16:43.893908 13486 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:36360:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 07:16:43.894567 13486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:16:43.898963 13624 tablet_service.cc:1467] Processing CreateTablet for tablet ddee14c0b6984f098e6addc6858e3226 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=899fd9ca82e74158aee6ae2ee150fcc8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:16:43.898952 13764 tablet_service.cc:1467] Processing CreateTablet for tablet ddee14c0b6984f098e6addc6858e3226 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=899fd9ca82e74158aee6ae2ee150fcc8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:16:43.899080 13904 tablet_service.cc:1467] Processing CreateTablet for tablet ddee14c0b6984f098e6addc6858e3226 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=899fd9ca82e74158aee6ae2ee150fcc8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:16:43.899672 13764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddee14c0b6984f098e6addc6858e3226. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:43.899672 13904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddee14c0b6984f098e6addc6858e3226. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:43.899680 13624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddee14c0b6984f098e6addc6858e3226. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:43.902125 13993 tablet_bootstrap.cc:492] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073: Bootstrap starting.
I20260620 07:16:43.902386 13995 tablet_bootstrap.cc:492] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: Bootstrap starting.
I20260620 07:16:43.903079 13993 tablet_bootstrap.cc:654] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:43.903182 13995 tablet_bootstrap.cc:654] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:43.903774 13994 tablet_bootstrap.cc:492] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09: Bootstrap starting.
I20260620 07:16:43.904167 13993 tablet_bootstrap.cc:492] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073: No bootstrap required, opened a new log
I20260620 07:16:43.904263 13993 ts_tablet_manager.cc:1397] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:16:43.904695 13993 raft_consensus.cc:348] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:43.904796 13994 tablet_bootstrap.cc:654] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:43.904821 13993 raft_consensus.cc:374] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:43.904853 13993 raft_consensus.cc:729] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974f9cbe41d64215837c494a7b9d7073, State: Initialized, Role: FOLLOWER
I20260620 07:16:43.904986 13993 consensus_queue.cc:260] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:43.905296 13993 ts_tablet_manager.cc:1428] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 07:16:43.905504 13999 tablet_replica.cc:439] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } }
I20260620 07:16:43.905642 13999 tablet_replica.cc:442] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:43.905947 13995 tablet_bootstrap.cc:492] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: No bootstrap required, opened a new log
I20260620 07:16:43.906046 13995 ts_tablet_manager.cc:1397] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 07:16:43.906242 13994 tablet_bootstrap.cc:492] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09: No bootstrap required, opened a new log
I20260620 07:16:43.906337 13994 ts_tablet_manager.cc:1397] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 07:16:43.906769 13994 raft_consensus.cc:348] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:43.906769 13995 raft_consensus.cc:348] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:43.906878 13995 raft_consensus.cc:374] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:43.906878 13994 raft_consensus.cc:374] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:43.906913 13995 raft_consensus.cc:729] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 340bbb5454bc4c09a445a2b88ba886bb, State: Initialized, Role: FOLLOWER
I20260620 07:16:43.906913 13994 raft_consensus.cc:729] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539f656f10e94f1080eec0cc58b15f09, State: Initialized, Role: FOLLOWER
I20260620 07:16:43.907039 13994 consensus_queue.cc:260] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:43.907039 13995 consensus_queue.cc:260] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:43.907294 13995 ts_tablet_manager.cc:1428] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 07:16:43.907280 14000 tablet_replica.cc:439] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } }
I20260620 07:16:43.907403 14000 tablet_replica.cc:442] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:43.907524 14001 tablet_replica.cc:439] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } }
I20260620 07:16:43.907649 14001 tablet_replica.cc:442] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:43.908299 13994 ts_tablet_manager.cc:1428] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 07:16:44.137409 14001 raft_consensus.cc:482] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:16:44.137576 14001 raft_consensus.cc:504] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:44.138711 14001 leader_election.cc:288] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129:42525), 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015)
I20260620 07:16:44.143301 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.139015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:45803 (local address 127.11.123.129:42525)
0620 07:16:44.139158 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:44.139162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:44.139296 (+ 134us) server_negotiation.cc:407] Connection header received
0620 07:16:44.139378 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:44.139382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:44.139434 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:44.139538 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:44.139849 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.140654 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.142127 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.142710 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.142832 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.142882 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:44.143105 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 07:16:44.143185 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:16:44.143314 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.138917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:42525 (local address 127.11.123.130:45803)
0620 07:16:44.139163 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:44.139178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:44.139285 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:44.139567 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:44.139571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:44.139591 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:44.139733 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:44.139742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.140785 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.140789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:44.142022 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:44.142034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.142856 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.142860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:44.142958 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.142996 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 07:16:44.143078 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:16:44.143146 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":57,"thread_start_us":95,"threads_started":1}
I20260620 07:16:44.143353 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.139298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:33015 (local address 127.11.123.130:48085)
0620 07:16:44.139610 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:44.139623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:44.139697 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:44.139949 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:44.139951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:44.139965 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:44.140055 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:44.140059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.140982 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.140987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:44.141956 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:44.141966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.142964 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.142972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:44.143071 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.143094 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:16:44.143162 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 07:16:44.143222 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":30,"thread_start_us":56,"threads_started":1}
I20260620 07:16:44.143702 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.139443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:48085 (local address 127.11.123.131:33015)
0620 07:16:44.139688 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:44.139691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:44.139703 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:16:44.139775 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:44.139777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:44.139844 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:44.139932 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:44.140153 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.140873 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.142081 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.142715 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.142832 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.142882 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:44.143455 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 07:16:44.143556 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260620 07:16:44.144114 13644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddee14c0b6984f098e6addc6858e3226" candidate_uuid: "340bbb5454bc4c09a445a2b88ba886bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974f9cbe41d64215837c494a7b9d7073" is_pre_election: true
I20260620 07:16:44.144145 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddee14c0b6984f098e6addc6858e3226" candidate_uuid: "340bbb5454bc4c09a445a2b88ba886bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539f656f10e94f1080eec0cc58b15f09" is_pre_election: true
I20260620 07:16:44.144266 13644 raft_consensus.cc:2454] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 340bbb5454bc4c09a445a2b88ba886bb in term 0.
I20260620 07:16:44.144338 13924 raft_consensus.cc:2454] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 340bbb5454bc4c09a445a2b88ba886bb in term 0.
I20260620 07:16:44.144635 13717 leader_election.cc:302] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340bbb5454bc4c09a445a2b88ba886bb, 974f9cbe41d64215837c494a7b9d7073; no voters:
I20260620 07:16:44.144901 14001 raft_consensus.cc:2781] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:16:44.144974 14001 raft_consensus.cc:482] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:16:44.145021 14001 raft_consensus.cc:3037] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:44.146124 14001 raft_consensus.cc:504] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:44.146468 14001 leader_election.cc:288] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [CANDIDATE]: Term 1 election: Requested vote from peers 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129:42525), 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015)
I20260620 07:16:44.146824 13644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddee14c0b6984f098e6addc6858e3226" candidate_uuid: "340bbb5454bc4c09a445a2b88ba886bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974f9cbe41d64215837c494a7b9d7073"
I20260620 07:16:44.146835 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddee14c0b6984f098e6addc6858e3226" candidate_uuid: "340bbb5454bc4c09a445a2b88ba886bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539f656f10e94f1080eec0cc58b15f09"
I20260620 07:16:44.146942 13924 raft_consensus.cc:3037] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:44.146945 13644 raft_consensus.cc:3037] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:44.147866 13644 raft_consensus.cc:2454] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 340bbb5454bc4c09a445a2b88ba886bb in term 1.
I20260620 07:16:44.148012 13924 raft_consensus.cc:2454] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 340bbb5454bc4c09a445a2b88ba886bb in term 1.
I20260620 07:16:44.148133 13717 leader_election.cc:302] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340bbb5454bc4c09a445a2b88ba886bb, 974f9cbe41d64215837c494a7b9d7073; no voters:
I20260620 07:16:44.148272 14001 raft_consensus.cc:2781] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:44.148448 14001 raft_consensus.cc:686] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [term 1 LEADER]: Becoming Leader. State: Replica: 340bbb5454bc4c09a445a2b88ba886bb, State: Running, Role: LEADER
I20260620 07:16:44.148756 14001 consensus_queue.cc:237] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:44.150321 14001 tablet_replica.cc:439] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: TxnStatusTablet state changed. Reason: New leader 340bbb5454bc4c09a445a2b88ba886bb. Latest consensus state: current_term: 1 leader_uuid: "340bbb5454bc4c09a445a2b88ba886bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } }
I20260620 07:16:44.150415 14001 tablet_replica.cc:442] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:16:44.150748 14019 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 07:16:44.150930 14019 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 07:16:44.151608 13486 catalog_manager.cc:5495] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130). New cstate: current_term: 1 leader_uuid: "340bbb5454bc4c09a445a2b88ba886bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } health_report { overall_health: UNKNOWN } } }
I20260620 07:16:44.198179 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.194448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:45059 (local address 127.0.0.1:33728)
0620 07:16:44.194627 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:44.194642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:44.194754 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:44.195089 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:44.195092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:44.195100 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:16:44.195209 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:44.195214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.195919 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.195924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:44.196521 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:44.196529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.196954 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.196957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:44.197050 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.197137 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:16:44.197876 (+ 739us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:16:44.197892 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 07:16:44.197959 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 07:16:44.198040 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 07:16:44.198230 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.194547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.11.123.130:45059)
0620 07:16:44.194664 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:44.194670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:44.194686 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:16:44.194906 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:44.194909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:44.194962 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:44.195079 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:44.195314 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.195810 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.196638 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.196846 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.196951 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.197287 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:44.197787 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:44.197904 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:44.197979 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 07:16:44.198036 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:16:44.203465 13924 raft_consensus.cc:1261] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Refusing update from remote peer 340bbb5454bc4c09a445a2b88ba886bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:44.203469 13644 raft_consensus.cc:1261] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Refusing update from remote peer 340bbb5454bc4c09a445a2b88ba886bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:44.204094 14001 consensus_queue.cc:1035] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:44.204293 14017 consensus_queue.cc:1035] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:44.207432 14000 tablet_replica.cc:439] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09: TxnStatusTablet state changed. Reason: New leader 340bbb5454bc4c09a445a2b88ba886bb. Latest consensus state: current_term: 1 leader_uuid: "340bbb5454bc4c09a445a2b88ba886bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } }
I20260620 07:16:44.207556 14000 tablet_replica.cc:442] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:44.209995 13999 tablet_replica.cc:439] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073: TxnStatusTablet state changed. Reason: New leader 340bbb5454bc4c09a445a2b88ba886bb. Latest consensus state: current_term: 1 leader_uuid: "340bbb5454bc4c09a445a2b88ba886bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } }
I20260620 07:16:44.210131 13999 tablet_replica.cc:442] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:44.211158 14001 tablet_replica.cc:439] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "340bbb5454bc4c09a445a2b88ba886bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } }
I20260620 07:16:44.211264 14001 tablet_replica.cc:442] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:16:44.211555 14024 mvcc.cc:205] Tried to move back new op lower bound from 7298825438010966016 to 7298825437796118528. Current Snapshot: MvccSnapshot[applied={T|T < 7298825438010966016}]
I20260620 07:16:44.211661 14000 tablet_replica.cc:439] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "340bbb5454bc4c09a445a2b88ba886bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } }
I20260620 07:16:44.211771 14000 tablet_replica.cc:442] T ddee14c0b6984f098e6addc6858e3226 P 539f656f10e94f1080eec0cc58b15f09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:44.214079 14017 tablet_replica.cc:439] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "340bbb5454bc4c09a445a2b88ba886bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } }
I20260620 07:16:44.214187 14017 tablet_replica.cc:442] T ddee14c0b6984f098e6addc6858e3226 P 340bbb5454bc4c09a445a2b88ba886bb: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:16:44.215312 13999 tablet_replica.cc:439] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "340bbb5454bc4c09a445a2b88ba886bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } } }
I20260620 07:16:44.215406 13999 tablet_replica.cc:442] T ddee14c0b6984f098e6addc6858e3226 P 974f9cbe41d64215837c494a7b9d7073: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:16:44.225682 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.221590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.11.123.131:33015)
0620 07:16:44.221711 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:44.221716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:44.221884 (+ 168us) server_negotiation.cc:407] Connection header received
0620 07:16:44.222002 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:44.222004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:44.222061 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:44.222160 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:44.222379 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.223159 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.223986 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.224335 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.224442 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.224665 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:44.225217 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:44.225374 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:44.225488 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 07:16:44.225546 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 07:16:44.236066 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.232634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:46773 (local address 127.11.123.130:45059)
0620 07:16:44.232764 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:44.232769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:44.232929 (+ 160us) server_negotiation.cc:407] Connection header received
0620 07:16:44.233054 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:44.233056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:44.233147 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:44.233220 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:44.233446 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.233931 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.235103 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.235311 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.235411 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.235701 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:44.235759 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:44.235810 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:44.235927 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 07:16:44.235965 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:16:44.236146 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.232549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:45059 (local address 127.11.123.131:46773)
0620 07:16:44.232863 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:44.232878 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:44.232976 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:44.233242 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:44.233245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:44.233253 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:16:44.233361 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:44.233366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.234023 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.234026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:44.234985 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:44.234995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.235427 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.235430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:44.235513 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.235561 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:16:44.235841 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:16:44.235847 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:16:44.235912 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 07:16:44.235989 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":107,"threads_started":1}
I20260620 07:16:44.243016 13644 raft_consensus.cc:1261] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Refusing update from remote peer 539f656f10e94f1080eec0cc58b15f09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:44.243180 13784 raft_consensus.cc:1261] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Refusing update from remote peer 539f656f10e94f1080eec0cc58b15f09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:44.243485 14003 consensus_queue.cc:1035] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:44.243641 14000 consensus_queue.cc:1035] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:44.249915 14023 mvcc.cc:205] Tried to move back new op lower bound from 7298825438175236096 to 7298825436570980352. Current Snapshot: MvccSnapshot[applied={T|T < 7298825438175236096}]
I20260620 07:16:44.267984 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.264198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:45311 (local address 127.11.123.131:33015)
0620 07:16:44.264297 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:44.264302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:44.264315 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:16:44.264403 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:44.264405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:44.264452 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:44.264528 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:44.264826 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.265663 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.266593 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.266943 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.267041 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.267437 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:44.267523 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:44.267611 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:44.267799 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 07:16:44.267869 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 07:16:44.267964 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.263882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:33015 (local address 127.11.123.130:45311)
0620 07:16:44.264201 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:44.264216 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:44.264303 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:44.264551 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:44.264554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:44.264563 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:16:44.264692 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:44.264699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.265793 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.265795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:44.266471 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:44.266480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.267138 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.267141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:44.267232 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.267279 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:16:44.267625 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:16:44.267633 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:16:44.267700 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 07:16:44.267788 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":102,"threads_started":1}
I20260620 07:16:44.290943 13487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:16:44.291136 13487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:16:44.293488 13487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511: Sending DeleteTablet for 3 replicas of tablet 135f9dd7f3d940c6ac5d358cb274ba7a
I20260620 07:16:44.294099 13904 tablet_service.cc:1514] Processing DeleteTablet for tablet 135f9dd7f3d940c6ac5d358cb274ba7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:44 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:38756
I20260620 07:16:44.294633 13624 tablet_service.cc:1514] Processing DeleteTablet for tablet 135f9dd7f3d940c6ac5d358cb274ba7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:44 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:56556
I20260620 07:16:44.295099 14044 tablet_replica.cc:330] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073: stopping tablet replica
I20260620 07:16:44.295104 13764 tablet_service.cc:1514] Processing DeleteTablet for tablet 135f9dd7f3d940c6ac5d358cb274ba7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:44 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:33718
I20260620 07:16:44.295548 14044 raft_consensus.cc:2229] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:16:44.295806 14044 raft_consensus.cc:2258] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:44.296281 13487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:16:44.296700 13487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:16:44.297588 14045 tablet_replica.cc:330] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb: stopping tablet replica
I20260620 07:16:44.297871 14045 raft_consensus.cc:2229] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:16:44.298038 14045 raft_consensus.cc:2258] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:44.299100 14044 ts_tablet_manager.cc:1905] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:44.302719 14044 ts_tablet_manager.cc:1918] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:16:44.302876 14044 log.cc:1196] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/wal/wals/135f9dd7f3d940c6ac5d358cb274ba7a
I20260620 07:16:44.303103 13624 tablet_service.cc:1467] Processing CreateTablet for tablet eb799a50914046709138085a987aa611 (DEFAULT_TABLE table=test-table [id=eb4603e7561d42a282ba71e95ed73d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:44.303359 13624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb799a50914046709138085a987aa611. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:44.303372 14044 ts_tablet_manager.cc:1939] T 135f9dd7f3d940c6ac5d358cb274ba7a P 974f9cbe41d64215837c494a7b9d7073: Deleting consensus metadata
I20260620 07:16:44.304737 13473 catalog_manager.cc:4841] TS 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129:42525): tablet 135f9dd7f3d940c6ac5d358cb274ba7a (table test-table [id=a7e842675f3841c2bf96f48bfc5d26db]) successfully deleted
I20260620 07:16:44.305344 13764 tablet_service.cc:1467] Processing CreateTablet for tablet eb799a50914046709138085a987aa611 (DEFAULT_TABLE table=test-table [id=eb4603e7561d42a282ba71e95ed73d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:44.305565 13993 tablet_bootstrap.cc:492] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073: Bootstrap starting.
I20260620 07:16:44.305624 13764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb799a50914046709138085a987aa611. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:44.306460 13993 tablet_bootstrap.cc:654] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:44.307842 13993 tablet_bootstrap.cc:492] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073: No bootstrap required, opened a new log
I20260620 07:16:44.307912 13993 ts_tablet_manager.cc:1397] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 07:16:44.308326 13904 tablet_service.cc:1467] Processing CreateTablet for tablet eb799a50914046709138085a987aa611 (DEFAULT_TABLE table=test-table [id=eb4603e7561d42a282ba71e95ed73d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:44.308324 13993 raft_consensus.cc:348] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:44.308430 13993 raft_consensus.cc:374] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:44.308462 13993 raft_consensus.cc:729] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974f9cbe41d64215837c494a7b9d7073, State: Initialized, Role: FOLLOWER
I20260620 07:16:44.308591 13904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb799a50914046709138085a987aa611. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:44.308580 13993 consensus_queue.cc:260] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:44.308830 13993 ts_tablet_manager.cc:1428] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 07:16:44.310969 13994 tablet_bootstrap.cc:492] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09: Bootstrap starting.
I20260620 07:16:44.311151 13995 tablet_bootstrap.cc:492] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb: Bootstrap starting.
I20260620 07:16:44.311906 13995 tablet_bootstrap.cc:654] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:44.311908 13994 tablet_bootstrap.cc:654] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:44.313859 14045 ts_tablet_manager.cc:1905] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:44.316394 14045 ts_tablet_manager.cc:1918] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:16:44.316510 14045 log.cc:1196] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/135f9dd7f3d940c6ac5d358cb274ba7a
I20260620 07:16:44.316776 14045 ts_tablet_manager.cc:1939] T 135f9dd7f3d940c6ac5d358cb274ba7a P 340bbb5454bc4c09a445a2b88ba886bb: Deleting consensus metadata
I20260620 07:16:44.317283 14043 tablet_replica.cc:330] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09: stopping tablet replica
I20260620 07:16:44.317629 14043 raft_consensus.cc:2229] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:16:44.317728 13995 tablet_bootstrap.cc:492] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb: No bootstrap required, opened a new log
I20260620 07:16:44.317827 13995 ts_tablet_manager.cc:1397] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 07:16:44.317965 13472 catalog_manager.cc:4841] TS 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130:45059): tablet 135f9dd7f3d940c6ac5d358cb274ba7a (table test-table [id=a7e842675f3841c2bf96f48bfc5d26db]) successfully deleted
I20260620 07:16:44.318198 13995 raft_consensus.cc:348] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:44.318305 13995 raft_consensus.cc:374] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:44.318338 13995 raft_consensus.cc:729] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 340bbb5454bc4c09a445a2b88ba886bb, State: Initialized, Role: FOLLOWER
I20260620 07:16:44.318454 13995 consensus_queue.cc:260] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:44.318621 14043 raft_consensus.cc:2258] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:44.318681 13995 ts_tablet_manager.cc:1428] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:16:44.320079 14043 ts_tablet_manager.cc:1905] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:44.322093 13994 tablet_bootstrap.cc:492] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09: No bootstrap required, opened a new log
I20260620 07:16:44.322207 13994 ts_tablet_manager.cc:1397] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20260620 07:16:44.322584 13994 raft_consensus.cc:348] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:44.322698 13994 raft_consensus.cc:374] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:44.322738 13994 raft_consensus.cc:729] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539f656f10e94f1080eec0cc58b15f09, State: Initialized, Role: FOLLOWER
I20260620 07:16:44.322878 13994 consensus_queue.cc:260] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:44.323175 13994 ts_tablet_manager.cc:1428] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:16:44.325642 14043 ts_tablet_manager.cc:1918] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:16:44.325734 14043 log.cc:1196] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/135f9dd7f3d940c6ac5d358cb274ba7a
I20260620 07:16:44.326078 14043 ts_tablet_manager.cc:1939] T 135f9dd7f3d940c6ac5d358cb274ba7a P 539f656f10e94f1080eec0cc58b15f09: Deleting consensus metadata
I20260620 07:16:44.327196 13473 catalog_manager.cc:4841] TS 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015): tablet 135f9dd7f3d940c6ac5d358cb274ba7a (table test-table [id=a7e842675f3841c2bf96f48bfc5d26db]) successfully deleted
I20260620 07:16:44.374269 14001 raft_consensus.cc:482] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:16:44.374393 14001 raft_consensus.cc:504] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:44.374763 14001 leader_election.cc:288] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129:42525), 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015)
I20260620 07:16:44.375244 13644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb799a50914046709138085a987aa611" candidate_uuid: "340bbb5454bc4c09a445a2b88ba886bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974f9cbe41d64215837c494a7b9d7073" is_pre_election: true
I20260620 07:16:44.375382 13644 raft_consensus.cc:2454] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 340bbb5454bc4c09a445a2b88ba886bb in term 0.
I20260620 07:16:44.375357 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb799a50914046709138085a987aa611" candidate_uuid: "340bbb5454bc4c09a445a2b88ba886bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539f656f10e94f1080eec0cc58b15f09" is_pre_election: true
I20260620 07:16:44.375510 13924 raft_consensus.cc:2454] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 340bbb5454bc4c09a445a2b88ba886bb in term 0.
I20260620 07:16:44.375677 13717 leader_election.cc:302] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340bbb5454bc4c09a445a2b88ba886bb, 974f9cbe41d64215837c494a7b9d7073; no voters:
I20260620 07:16:44.375851 14001 raft_consensus.cc:2781] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:16:44.375895 14001 raft_consensus.cc:482] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:16:44.375927 14001 raft_consensus.cc:3037] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:44.377107 14001 raft_consensus.cc:504] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:44.377404 14001 leader_election.cc:288] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [CANDIDATE]: Term 1 election: Requested vote from peers 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129:42525), 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015)
I20260620 07:16:44.377656 13644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb799a50914046709138085a987aa611" candidate_uuid: "340bbb5454bc4c09a445a2b88ba886bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974f9cbe41d64215837c494a7b9d7073"
I20260620 07:16:44.377750 13644 raft_consensus.cc:3037] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:44.377751 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb799a50914046709138085a987aa611" candidate_uuid: "340bbb5454bc4c09a445a2b88ba886bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539f656f10e94f1080eec0cc58b15f09"
I20260620 07:16:44.377846 13924 raft_consensus.cc:3037] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:44.378726 13644 raft_consensus.cc:2454] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 340bbb5454bc4c09a445a2b88ba886bb in term 1.
I20260620 07:16:44.378886 13924 raft_consensus.cc:2454] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 340bbb5454bc4c09a445a2b88ba886bb in term 1.
I20260620 07:16:44.378999 13717 leader_election.cc:302] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340bbb5454bc4c09a445a2b88ba886bb, 974f9cbe41d64215837c494a7b9d7073; no voters:
I20260620 07:16:44.379144 14001 raft_consensus.cc:2781] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:44.379206 14001 raft_consensus.cc:686] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 1 LEADER]: Becoming Leader. State: Replica: 340bbb5454bc4c09a445a2b88ba886bb, State: Running, Role: LEADER
I20260620 07:16:44.379323 14001 consensus_queue.cc:237] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:44.380975 13487 catalog_manager.cc:5495] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130). New cstate: current_term: 1 leader_uuid: "340bbb5454bc4c09a445a2b88ba886bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } health_report { overall_health: UNKNOWN } } }
I20260620 07:16:44.404722 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.401384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.11.123.130:45059)
0620 07:16:44.401499 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:44.401504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:44.401522 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:16:44.401646 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:44.401650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:44.401714 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:44.401820 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:44.401980 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.402764 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.403532 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.403879 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.403985 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.404252 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:44.404340 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:44.404427 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:44.404544 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 07:16:44.404594 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:16:44.413518 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.410134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:45059 (local address 127.11.123.130:46701)
0620 07:16:44.410257 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:44.410269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:44.410330 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:44.410740 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:44.410743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:44.410749 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:16:44.410824 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:44.410828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.411736 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.411738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:44.412284 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:44.412291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.412819 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.412822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:44.412880 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.412903 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:16:44.413197 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:16:44.413202 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:16:44.413255 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 07:16:44.413380 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":2}
I20260620 07:16:44.413534 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:44.410420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:46701 (local address 127.11.123.130:45059)
0620 07:16:44.410535 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:44.410539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:44.410552 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:16:44.410592 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:44.410594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:44.410640 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:44.410721 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:44.410917 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.411640 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.412394 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:44.412712 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:44.412825 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:44.413010 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:44.413118 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:44.413195 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:44.413283 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 07:16:44.413378 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":43}
I20260620 07:16:44.418713 13924 raft_consensus.cc:1261] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Refusing update from remote peer 340bbb5454bc4c09a445a2b88ba886bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:44.418926 13644 raft_consensus.cc:1261] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Refusing update from remote peer 340bbb5454bc4c09a445a2b88ba886bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:44.419209 14001 consensus_queue.cc:1035] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:44.419342 14017 consensus_queue.cc:1035] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:16:44.435314 14022 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 07:16:45.990240 13487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:16:45.990392 13487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:16:45.992666 13487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511: Sending DeleteTablet for 3 replicas of tablet eb799a50914046709138085a987aa611
I20260620 07:16:45.993276 13904 tablet_service.cc:1514] Processing DeleteTablet for tablet eb799a50914046709138085a987aa611 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:45 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:38756
I20260620 07:16:45.993261 13763 tablet_service.cc:1514] Processing DeleteTablet for tablet eb799a50914046709138085a987aa611 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:45 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:33718
I20260620 07:16:45.993314 13624 tablet_service.cc:1514] Processing DeleteTablet for tablet eb799a50914046709138085a987aa611 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:45 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:56556
I20260620 07:16:45.993676 14073 tablet_replica.cc:330] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09: stopping tablet replica
I20260620 07:16:45.993896 14073 raft_consensus.cc:2229] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:16:45.993590 14074 tablet_replica.cc:330] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb: stopping tablet replica
I20260620 07:16:45.994093 14073 raft_consensus.cc:2258] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:45.994525 14074 raft_consensus.cc:2229] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:16:45.994792 14073 ts_tablet_manager.cc:1905] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:45.994880 14074 raft_consensus.cc:2258] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:45.995586 14074 ts_tablet_manager.cc:1905] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:45.996876 14075 tablet_replica.cc:330] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073: stopping tablet replica
I20260620 07:16:45.997035 14075 raft_consensus.cc:2229] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:16:45.997231 14075 raft_consensus.cc:2258] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:45.997639 14073 ts_tablet_manager.cc:1918] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:16:45.997758 14073 log.cc:1196] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/eb799a50914046709138085a987aa611
I20260620 07:16:45.998016 14073 ts_tablet_manager.cc:1939] T eb799a50914046709138085a987aa611 P 539f656f10e94f1080eec0cc58b15f09: Deleting consensus metadata
I20260620 07:16:45.998404 14074 ts_tablet_manager.cc:1918] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:16:45.998492 14074 log.cc:1196] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/eb799a50914046709138085a987aa611
I20260620 07:16:45.998659 13473 catalog_manager.cc:4841] TS 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015): tablet eb799a50914046709138085a987aa611 (table test-table [id=eb4603e7561d42a282ba71e95ed73d50]) successfully deleted
I20260620 07:16:45.998793 14074 ts_tablet_manager.cc:1939] T eb799a50914046709138085a987aa611 P 340bbb5454bc4c09a445a2b88ba886bb: Deleting consensus metadata
I20260620 07:16:45.999442 14075 ts_tablet_manager.cc:1905] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:45.999485 13472 catalog_manager.cc:4841] TS 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130:45059): tablet eb799a50914046709138085a987aa611 (table test-table [id=eb4603e7561d42a282ba71e95ed73d50]) successfully deleted
I20260620 07:16:46.002346 14075 ts_tablet_manager.cc:1918] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:16:46.002435 14075 log.cc:1196] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/wal/wals/eb799a50914046709138085a987aa611
I20260620 07:16:46.002707 14075 ts_tablet_manager.cc:1939] T eb799a50914046709138085a987aa611 P 974f9cbe41d64215837c494a7b9d7073: Deleting consensus metadata
I20260620 07:16:46.003485 13473 catalog_manager.cc:4841] TS 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129:42525): tablet eb799a50914046709138085a987aa611 (table test-table [id=eb4603e7561d42a282ba71e95ed73d50]) successfully deleted
Jun 20 07:16:46 dist-test-slave-j663 krb5kdc[13442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:16:46 dist-test-slave-j663 krb5kdc[13442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:46.024622 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:46.015627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36364 (local address 127.11.123.190:37289)
0620 07:16:46.015944 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:46.015954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:46.015970 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:16:46.016129 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:46.016132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:46.016184 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:46.016279 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:46.016983 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.017606 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.018437 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.018671 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.018775 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:46.021511 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:46.021514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:46.021516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:46.021541 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:46.022965 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:46.023385 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:46.023390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:46.023392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:46.023476 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:46.023744 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:46.023748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:46.023750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:46.023942 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:46.024083 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:46.024313 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 07:16:46.024442 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":112,"threads_started":1}
I20260620 07:16:46.027352 13487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:16:46.027820 13487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:16:46.032670 13624 tablet_service.cc:1467] Processing CreateTablet for tablet c202832dda354c98bf09cecb9e0ebfee (DEFAULT_TABLE table=test-table [id=3749fad901104af0b55921b1ed9e834e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:46.032759 13763 tablet_service.cc:1467] Processing CreateTablet for tablet c202832dda354c98bf09cecb9e0ebfee (DEFAULT_TABLE table=test-table [id=3749fad901104af0b55921b1ed9e834e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:46.033002 13624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c202832dda354c98bf09cecb9e0ebfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:46.033001 13763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c202832dda354c98bf09cecb9e0ebfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:46.032960 13904 tablet_service.cc:1467] Processing CreateTablet for tablet c202832dda354c98bf09cecb9e0ebfee (DEFAULT_TABLE table=test-table [id=3749fad901104af0b55921b1ed9e834e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:46.033255 13904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c202832dda354c98bf09cecb9e0ebfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:46.035770 14085 tablet_bootstrap.cc:492] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073: Bootstrap starting.
I20260620 07:16:46.035770 14086 tablet_bootstrap.cc:492] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb: Bootstrap starting.
I20260620 07:16:46.035872 14084 tablet_bootstrap.cc:492] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09: Bootstrap starting.
I20260620 07:16:46.036582 14084 tablet_bootstrap.cc:654] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:46.036615 14086 tablet_bootstrap.cc:654] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:46.036615 14085 tablet_bootstrap.cc:654] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:46.037686 14086 tablet_bootstrap.cc:492] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb: No bootstrap required, opened a new log
I20260620 07:16:46.037737 14084 tablet_bootstrap.cc:492] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09: No bootstrap required, opened a new log
I20260620 07:16:46.037737 14085 tablet_bootstrap.cc:492] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073: No bootstrap required, opened a new log
I20260620 07:16:46.037796 14086 ts_tablet_manager.cc:1397] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 07:16:46.037811 14085 ts_tablet_manager.cc:1397] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 07:16:46.037811 14084 ts_tablet_manager.cc:1397] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:16:46.038254 14084 raft_consensus.cc:348] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.038259 14086 raft_consensus.cc:348] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.038373 14086 raft_consensus.cc:374] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:46.038372 14084 raft_consensus.cc:374] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:46.038420 14084 raft_consensus.cc:729] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539f656f10e94f1080eec0cc58b15f09, State: Initialized, Role: FOLLOWER
I20260620 07:16:46.038414 14086 raft_consensus.cc:729] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 340bbb5454bc4c09a445a2b88ba886bb, State: Initialized, Role: FOLLOWER
I20260620 07:16:46.038499 14085 raft_consensus.cc:348] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.038594 14085 raft_consensus.cc:374] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:46.038548 14084 consensus_queue.cc:260] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.038546 14086 consensus_queue.cc:260] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.038625 14085 raft_consensus.cc:729] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974f9cbe41d64215837c494a7b9d7073, State: Initialized, Role: FOLLOWER
I20260620 07:16:46.038771 14085 consensus_queue.cc:260] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.038846 14086 ts_tablet_manager.cc:1428] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:16:46.038911 14084 ts_tablet_manager.cc:1428] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:16:46.039136 14085 ts_tablet_manager.cc:1428] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260620 07:16:46.076434 14091 raft_consensus.cc:482] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:16:46.076640 14091 raft_consensus.cc:504] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.077844 14091 leader_election.cc:288] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130:45059), 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015)
I20260620 07:16:46.081974 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:46.078121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:33015 (local address 127.11.123.129:46423)
0620 07:16:46.078443 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:46.078458 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:46.078571 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:46.078865 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:46.078867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:46.078903 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:46.079047 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:46.079051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.079809 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.079812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:46.080686 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:46.080695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.081518 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.081522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:46.081620 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:46.081642 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:16:46.081704 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 07:16:46.081774 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":21,"spinlock_wait_cycles":2688,"thread_start_us":107,"threads_started":1}
I20260620 07:16:46.081983 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:46.078027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:45059 (local address 127.11.123.129:39363)
0620 07:16:46.078425 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:46.078441 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:46.078554 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:46.078830 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:46.078833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:46.078849 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:16:46.079029 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:46.079035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.079793 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.079797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:46.080697 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:46.080706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.081638 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.081641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:46.081735 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:46.081753 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 07:16:46.081811 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 07:16:46.081865 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":81,"spinlock_wait_cycles":24960,"thread_start_us":141,"threads_started":1}
I20260620 07:16:46.082126 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:46.078220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:46423 (local address 127.11.123.131:33015)
0620 07:16:46.078488 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:46.078498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:46.078514 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:16:46.078699 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:46.078701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:46.078742 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:46.078838 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:46.079176 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.079680 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.080806 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.081390 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.081788 (+ 398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:46.081847 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:46.081884 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 07:16:46.081956 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
I20260620 07:16:46.082453 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c202832dda354c98bf09cecb9e0ebfee" candidate_uuid: "974f9cbe41d64215837c494a7b9d7073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539f656f10e94f1080eec0cc58b15f09" is_pre_election: true
I20260620 07:16:46.082602 13924 raft_consensus.cc:2454] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974f9cbe41d64215837c494a7b9d7073 in term 0.
I20260620 07:16:46.082696 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:46.078140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:39363 (local address 127.11.123.130:45059)
0620 07:16:46.078460 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:46.078471 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:46.078493 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:16:46.078668 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:46.078674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:46.078724 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:46.078789 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:16:46.079136 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.079675 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.080848 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.081402 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.081531 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:46.081589 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:46.082031 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 07:16:46.082555 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":98,"threads_started":1}
I20260620 07:16:46.083017 13577 leader_election.cc:302] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 539f656f10e94f1080eec0cc58b15f09, 974f9cbe41d64215837c494a7b9d7073; no voters:
I20260620 07:16:46.083057 13784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c202832dda354c98bf09cecb9e0ebfee" candidate_uuid: "974f9cbe41d64215837c494a7b9d7073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340bbb5454bc4c09a445a2b88ba886bb" is_pre_election: true
I20260620 07:16:46.083179 13784 raft_consensus.cc:2454] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974f9cbe41d64215837c494a7b9d7073 in term 0.
I20260620 07:16:46.083300 14091 raft_consensus.cc:2781] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:16:46.083377 14091 raft_consensus.cc:482] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:16:46.083420 14091 raft_consensus.cc:3037] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:46.084398 14091 raft_consensus.cc:504] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.084686 14091 leader_election.cc:288] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [CANDIDATE]: Term 1 election: Requested vote from peers 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130:45059), 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015)
I20260620 07:16:46.085000 13784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c202832dda354c98bf09cecb9e0ebfee" candidate_uuid: "974f9cbe41d64215837c494a7b9d7073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340bbb5454bc4c09a445a2b88ba886bb"
I20260620 07:16:46.085000 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c202832dda354c98bf09cecb9e0ebfee" candidate_uuid: "974f9cbe41d64215837c494a7b9d7073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539f656f10e94f1080eec0cc58b15f09"
I20260620 07:16:46.085167 13784 raft_consensus.cc:3037] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:46.085171 13924 raft_consensus.cc:3037] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:46.086057 13784 raft_consensus.cc:2454] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974f9cbe41d64215837c494a7b9d7073 in term 1.
I20260620 07:16:46.086056 13924 raft_consensus.cc:2454] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974f9cbe41d64215837c494a7b9d7073 in term 1.
I20260620 07:16:46.086333 13576 leader_election.cc:302] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340bbb5454bc4c09a445a2b88ba886bb, 974f9cbe41d64215837c494a7b9d7073; no voters:
I20260620 07:16:46.086524 14091 raft_consensus.cc:2781] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:46.086766 14091 raft_consensus.cc:686] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 1 LEADER]: Becoming Leader. State: Replica: 974f9cbe41d64215837c494a7b9d7073, State: Running, Role: LEADER
I20260620 07:16:46.087011 14091 consensus_queue.cc:237] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.089067 13484 catalog_manager.cc:5495] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129). New cstate: current_term: 1 leader_uuid: "974f9cbe41d64215837c494a7b9d7073" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } health_report { overall_health: UNKNOWN } } }
I20260620 07:16:46.106870 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:46.103373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.11.123.129:42525)
0620 07:16:46.103631 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:46.103638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:46.103649 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:16:46.103714 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:46.103715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:46.103809 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:46.103910 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:46.104116 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.104630 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.105518 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.105745 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.105824 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:46.106084 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:46.106527 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:46.106613 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:46.106697 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 07:16:46.106746 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":129,"threads_started":1}
I20260620 07:16:46.116439 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:46.113169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:57153 (local address 127.11.123.130:45059)
0620 07:16:46.113280 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:46.113283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:46.113406 (+ 123us) server_negotiation.cc:407] Connection header received
0620 07:16:46.113464 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:46.113472 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:46.113514 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:46.113610 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:46.113891 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.114422 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.115429 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.115649 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.115760 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:46.116025 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:46.116094 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:46.116165 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:46.116258 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 07:16:46.116305 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 07:16:46.116478 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:46.112984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:45059 (local address 127.11.123.129:57153)
0620 07:16:46.113308 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:46.113320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:46.113403 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:46.113632 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:46.113634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:46.113643 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:16:46.113768 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:46.113774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.114549 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.114552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:46.115298 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:46.115307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.115767 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.115770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:46.115860 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:46.115900 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:16:46.116181 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:16:46.116189 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:16:46.116248 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 07:16:46.116324 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":120,"threads_started":1}
I20260620 07:16:46.122074 13784 raft_consensus.cc:1261] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Refusing update from remote peer 974f9cbe41d64215837c494a7b9d7073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:46.122092 13924 raft_consensus.cc:1261] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Refusing update from remote peer 974f9cbe41d64215837c494a7b9d7073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:46.122511 14091 consensus_queue.cc:1035] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:46.122737 14096 consensus_queue.cc:1035] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:46.125838 14054 mvcc.cc:205] Tried to move back new op lower bound from 7298825445872336896 to 7298825445734588416. Current Snapshot: MvccSnapshot[applied={T|T < 7298825445872336896}]
I20260620 07:16:46.138571 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:46.135080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:57423 (local address 127.11.123.129:42525)
0620 07:16:46.135419 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:46.135424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:46.135438 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:16:46.135479 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:46.135481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:46.135528 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:46.135609 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:46.135942 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.136452 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.137376 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.137669 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.137771 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:46.138055 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:46.138165 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:46.138292 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:46.138345 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 07:16:46.138395 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
I20260620 07:16:46.138609 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:46.135017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:42525 (local address 127.11.123.130:57423)
0620 07:16:46.135255 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:46.135271 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:46.135367 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:46.135709 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:46.135712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:46.135720 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:16:46.135832 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:46.135837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.136556 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.136561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:46.137253 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:46.137263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:46.137769 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:46.137773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:46.137862 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:46.137910 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:16:46.138257 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:16:46.138264 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:16:46.138339 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 07:16:46.138444 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260620 07:16:46.165436 13485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:16:46.165606 13485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:16:46.167689 13485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511: Sending DeleteTablet for 3 replicas of tablet c202832dda354c98bf09cecb9e0ebfee
I20260620 07:16:46.168287 13763 tablet_service.cc:1514] Processing DeleteTablet for tablet c202832dda354c98bf09cecb9e0ebfee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:46 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:33718
I20260620 07:16:46.168435 13624 tablet_service.cc:1514] Processing DeleteTablet for tablet c202832dda354c98bf09cecb9e0ebfee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:46 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:56556
I20260620 07:16:46.168617 14075 tablet_replica.cc:330] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073: stopping tablet replica
I20260620 07:16:46.168694 13904 tablet_service.cc:1514] Processing DeleteTablet for tablet c202832dda354c98bf09cecb9e0ebfee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:46 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:38756
I20260620 07:16:46.168730 14075 raft_consensus.cc:2229] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:16:46.168905 14074 tablet_replica.cc:330] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb: stopping tablet replica
I20260620 07:16:46.169020 14075 raft_consensus.cc:2258] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:46.169039 14074 raft_consensus.cc:2229] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:16:46.169209 14074 raft_consensus.cc:2258] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:46.169451 14073 tablet_replica.cc:330] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09: stopping tablet replica
I20260620 07:16:46.169586 14073 raft_consensus.cc:2229] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:16:46.169698 14073 raft_consensus.cc:2258] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:46.169773 14075 ts_tablet_manager.cc:1905] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:46.170419 14074 ts_tablet_manager.cc:1905] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:46.170513 14073 ts_tablet_manager.cc:1905] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:46.172405 14075 ts_tablet_manager.cc:1918] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:16:46.172470 14075 log.cc:1196] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-0/wal/wals/c202832dda354c98bf09cecb9e0ebfee
I20260620 07:16:46.172726 14075 ts_tablet_manager.cc:1939] T c202832dda354c98bf09cecb9e0ebfee P 974f9cbe41d64215837c494a7b9d7073: Deleting consensus metadata
I20260620 07:16:46.172991 14074 ts_tablet_manager.cc:1918] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:16:46.173030 14073 ts_tablet_manager.cc:1918] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:16:46.173326 14073 log.cc:1196] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/c202832dda354c98bf09cecb9e0ebfee
I20260620 07:16:46.173470 13473 catalog_manager.cc:4841] TS 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129:42525): tablet c202832dda354c98bf09cecb9e0ebfee (table test-table [id=3749fad901104af0b55921b1ed9e834e]) successfully deleted
I20260620 07:16:46.173429 13485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 07:16:46.173601 14073 ts_tablet_manager.cc:1939] T c202832dda354c98bf09cecb9e0ebfee P 539f656f10e94f1080eec0cc58b15f09: Deleting consensus metadata
W20260620 07:16:46.173866 13485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:16:46.173058 14074 log.cc:1196] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/c202832dda354c98bf09cecb9e0ebfee
I20260620 07:16:46.174194 13473 catalog_manager.cc:4841] TS 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015): tablet c202832dda354c98bf09cecb9e0ebfee (table test-table [id=3749fad901104af0b55921b1ed9e834e]) successfully deleted
I20260620 07:16:46.174314 14074 ts_tablet_manager.cc:1939] T c202832dda354c98bf09cecb9e0ebfee P 340bbb5454bc4c09a445a2b88ba886bb: Deleting consensus metadata
I20260620 07:16:46.174881 13472 catalog_manager.cc:4841] TS 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130:45059): tablet c202832dda354c98bf09cecb9e0ebfee (table test-table [id=3749fad901104af0b55921b1ed9e834e]) successfully deleted
I20260620 07:16:46.178416 13624 tablet_service.cc:1467] Processing CreateTablet for tablet b39a3468a83c49b79abdcb1ff61d329b (DEFAULT_TABLE table=test-table [id=66904b60a41d4bad99046804aab975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:46.178599 13763 tablet_service.cc:1467] Processing CreateTablet for tablet b39a3468a83c49b79abdcb1ff61d329b (DEFAULT_TABLE table=test-table [id=66904b60a41d4bad99046804aab975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:46.178599 13904 tablet_service.cc:1467] Processing CreateTablet for tablet b39a3468a83c49b79abdcb1ff61d329b (DEFAULT_TABLE table=test-table [id=66904b60a41d4bad99046804aab975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:16:46.178676 13624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39a3468a83c49b79abdcb1ff61d329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:46.178820 13763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39a3468a83c49b79abdcb1ff61d329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:46.178820 13904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39a3468a83c49b79abdcb1ff61d329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:46.180905 14086 tablet_bootstrap.cc:492] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb: Bootstrap starting.
I20260620 07:16:46.180977 14084 tablet_bootstrap.cc:492] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09: Bootstrap starting.
I20260620 07:16:46.181551 14085 tablet_bootstrap.cc:492] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073: Bootstrap starting.
I20260620 07:16:46.181774 14086 tablet_bootstrap.cc:654] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:46.181774 14084 tablet_bootstrap.cc:654] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:46.182425 14085 tablet_bootstrap.cc:654] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:46.183041 14084 tablet_bootstrap.cc:492] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09: No bootstrap required, opened a new log
I20260620 07:16:46.183041 14086 tablet_bootstrap.cc:492] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb: No bootstrap required, opened a new log
I20260620 07:16:46.183123 14086 ts_tablet_manager.cc:1397] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:46.183123 14084 ts_tablet_manager.cc:1397] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:16:46.183490 14084 raft_consensus.cc:348] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.183490 14086 raft_consensus.cc:348] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.183602 14086 raft_consensus.cc:374] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:46.183602 14084 raft_consensus.cc:374] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:46.183637 14086 raft_consensus.cc:729] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 340bbb5454bc4c09a445a2b88ba886bb, State: Initialized, Role: FOLLOWER
I20260620 07:16:46.183637 14084 raft_consensus.cc:729] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539f656f10e94f1080eec0cc58b15f09, State: Initialized, Role: FOLLOWER
I20260620 07:16:46.183754 14084 consensus_queue.cc:260] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.183755 14086 consensus_queue.cc:260] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.184008 14084 ts_tablet_manager.cc:1428] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:16:46.184021 14086 ts_tablet_manager.cc:1428] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:16:46.184834 14085 tablet_bootstrap.cc:492] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073: No bootstrap required, opened a new log
I20260620 07:16:46.184918 14085 ts_tablet_manager.cc:1397] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 07:16:46.185365 14085 raft_consensus.cc:348] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.185472 14085 raft_consensus.cc:374] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:46.185520 14085 raft_consensus.cc:729] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974f9cbe41d64215837c494a7b9d7073, State: Initialized, Role: FOLLOWER
I20260620 07:16:46.185621 14085 consensus_queue.cc:260] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.185871 14085 ts_tablet_manager.cc:1428] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:16:46.196069 14096 raft_consensus.cc:482] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:16:46.196221 14096 raft_consensus.cc:504] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.196642 14096 leader_election.cc:288] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130:45059), 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015)
I20260620 07:16:46.196985 13784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39a3468a83c49b79abdcb1ff61d329b" candidate_uuid: "974f9cbe41d64215837c494a7b9d7073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340bbb5454bc4c09a445a2b88ba886bb" is_pre_election: true
I20260620 07:16:46.197000 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39a3468a83c49b79abdcb1ff61d329b" candidate_uuid: "974f9cbe41d64215837c494a7b9d7073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539f656f10e94f1080eec0cc58b15f09" is_pre_election: true
I20260620 07:16:46.197146 13784 raft_consensus.cc:2454] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974f9cbe41d64215837c494a7b9d7073 in term 0.
I20260620 07:16:46.197168 13924 raft_consensus.cc:2454] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974f9cbe41d64215837c494a7b9d7073 in term 0.
I20260620 07:16:46.197423 13576 leader_election.cc:302] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340bbb5454bc4c09a445a2b88ba886bb, 974f9cbe41d64215837c494a7b9d7073; no voters:
I20260620 07:16:46.197623 14096 raft_consensus.cc:2781] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:16:46.197688 14096 raft_consensus.cc:482] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:16:46.197737 14096 raft_consensus.cc:3037] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:46.198974 14096 raft_consensus.cc:504] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.199340 14096 leader_election.cc:288] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [CANDIDATE]: Term 1 election: Requested vote from peers 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130:45059), 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015)
I20260620 07:16:46.199605 13784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39a3468a83c49b79abdcb1ff61d329b" candidate_uuid: "974f9cbe41d64215837c494a7b9d7073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340bbb5454bc4c09a445a2b88ba886bb"
I20260620 07:16:46.199643 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39a3468a83c49b79abdcb1ff61d329b" candidate_uuid: "974f9cbe41d64215837c494a7b9d7073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539f656f10e94f1080eec0cc58b15f09"
I20260620 07:16:46.199740 13784 raft_consensus.cc:3037] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:46.199748 13924 raft_consensus.cc:3037] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:46.200609 13784 raft_consensus.cc:2454] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974f9cbe41d64215837c494a7b9d7073 in term 1.
I20260620 07:16:46.200827 13924 raft_consensus.cc:2454] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974f9cbe41d64215837c494a7b9d7073 in term 1.
I20260620 07:16:46.200906 13576 leader_election.cc:302] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340bbb5454bc4c09a445a2b88ba886bb, 974f9cbe41d64215837c494a7b9d7073; no voters:
I20260620 07:16:46.201097 14096 raft_consensus.cc:2781] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:46.201189 14096 raft_consensus.cc:686] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [term 1 LEADER]: Becoming Leader. State: Replica: 974f9cbe41d64215837c494a7b9d7073, State: Running, Role: LEADER
I20260620 07:16:46.201329 14096 consensus_queue.cc:237] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } }
I20260620 07:16:46.202829 13485 catalog_manager.cc:5495] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129). New cstate: current_term: 1 leader_uuid: "974f9cbe41d64215837c494a7b9d7073" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974f9cbe41d64215837c494a7b9d7073" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 42525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 } health_report { overall_health: UNKNOWN } } }
I20260620 07:16:46.222074 13924 raft_consensus.cc:1261] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Refusing update from remote peer 974f9cbe41d64215837c494a7b9d7073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:46.222112 13784 raft_consensus.cc:1261] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Refusing update from remote peer 974f9cbe41d64215837c494a7b9d7073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:46.222517 14105 consensus_queue.cc:1035] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "340bbb5454bc4c09a445a2b88ba886bb" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:46.222674 14096 consensus_queue.cc:1035] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "539f656f10e94f1080eec0cc58b15f09" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:16:46.237316 14058 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 07:16:47.787344 13485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:16:47.787495 13485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:16:47.789760 13485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9934c91f29c54d698951790fb7e21511: Sending DeleteTablet for 3 replicas of tablet b39a3468a83c49b79abdcb1ff61d329b
I20260620 07:16:47.790310 13624 tablet_service.cc:1514] Processing DeleteTablet for tablet b39a3468a83c49b79abdcb1ff61d329b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:47 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:56556
I20260620 07:16:47.790331 13904 tablet_service.cc:1514] Processing DeleteTablet for tablet b39a3468a83c49b79abdcb1ff61d329b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:47 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:38756
I20260620 07:16:47.790346 13763 tablet_service.cc:1514] Processing DeleteTablet for tablet b39a3468a83c49b79abdcb1ff61d329b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:16:47 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:33718
I20260620 07:16:47.790679 14136 tablet_replica.cc:330] T b39a3468a83c49b79abdcb1ff61d329b P 974f9cbe41d64215837c494a7b9d7073: stopping tablet replica
I20260620 07:16:47.790683 14138 tablet_replica.cc:330] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb: stopping tablet replica
I20260620 07:16:47.790728 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 13559
I20260620 07:16:47.791790 14137 tablet_replica.cc:330] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09: stopping tablet replica
I20260620 07:16:47.791883 14137 raft_consensus.cc:2229] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:16:47.792014 14137 raft_consensus.cc:2258] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:47.792604 14137 ts_tablet_manager.cc:1905] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:47.790840 14138 raft_consensus.cc:2229] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:16:47.792838 14138 raft_consensus.cc:2258] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:16:47.793499 14138 ts_tablet_manager.cc:1905] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:16:47.794831 14137 ts_tablet_manager.cc:1918] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:16:47.794929 14137 log.cc:1196] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/b39a3468a83c49b79abdcb1ff61d329b
I20260620 07:16:47.795176 14137 ts_tablet_manager.cc:1939] T b39a3468a83c49b79abdcb1ff61d329b P 539f656f10e94f1080eec0cc58b15f09: Deleting consensus metadata
I20260620 07:16:47.795768 13473 catalog_manager.cc:4841] TS 539f656f10e94f1080eec0cc58b15f09 (127.11.123.131:33015): tablet b39a3468a83c49b79abdcb1ff61d329b (table test-table [id=66904b60a41d4bad99046804aab975d1]) successfully deleted
I20260620 07:16:47.796712 14138 ts_tablet_manager.cc:1918] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:16:47.796804 14138 log.cc:1196] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/b39a3468a83c49b79abdcb1ff61d329b
I20260620 07:16:47.797116 14138 ts_tablet_manager.cc:1939] T b39a3468a83c49b79abdcb1ff61d329b P 340bbb5454bc4c09a445a2b88ba886bb: Deleting consensus metadata
I20260620 07:16:47.797806 13472 catalog_manager.cc:4841] TS 340bbb5454bc4c09a445a2b88ba886bb (127.11.123.130:45059): tablet b39a3468a83c49b79abdcb1ff61d329b (table test-table [id=66904b60a41d4bad99046804aab975d1]) successfully deleted
W20260620 07:16:47.799357 13473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:42525 (error 108)
I20260620 07:16:47.799612 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 13699
I20260620 07:16:47.801151 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:47.800493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:42525 (local address 127.0.0.1:56586)
0620 07:16:47.800887 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:47.800980 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":66,"threads_started":1}
W20260620 07:16:47.801294 13473 catalog_manager.cc:4584] TS 974f9cbe41d64215837c494a7b9d7073 (127.11.123.129:42525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b39a3468a83c49b79abdcb1ff61d329b: Network error: Client connection negotiation failed: client connection to 127.11.123.129:42525: connect: Connection refused (error 111)
I20260620 07:16:47.807499 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 13839
I20260620 07:16:47.813858 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 13455
2026-06-20T07:16:47Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6263 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:47 dist-test-slave-j663 krb5kdc[14143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:47 dist-test-slave-j663 krb5kdc[14143](info): set up 2 sockets
Jun 20 07:16:47 dist-test-slave-j663 krb5kdc[14143](info): commencing operation
krb5kdc: starting...
W20260620 07:16:48.889065 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:16:48 dist-test-slave-j663 krb5kdc[14143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:16:49.010316 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:39325
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:39325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:49.116832 14156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:49.117126 14156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:49.117201 14156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:49.121198 14156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:49.121272 14156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:49.121309 14156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:49.121345 14156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:49.121381 14156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:49.126406 14156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:39325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:39325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:49.127377 14156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:49.128347 14156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:49.136837 14164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:49.136812 14161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:49.136873 14156 server_base.cc:1034] running on GCE node
W20260620 07:16:49.136822 14162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:49.137578 14156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:49.138612 14156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:49.139765 14156 hybrid_clock.cc:648] HybridClock initialized: now 1781939809139733 us; error 44 us; skew 500 ppm
Jun 20 07:16:49 dist-test-slave-j663 krb5kdc[14143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939809, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:49.142405 14156 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:16:49.143251 14156 webserver.cc:460] Webserver started at http://127.11.123.190:38431/ using document root <none> and password file <none>
I20260620 07:16:49.143777 14156 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:49.143836 14156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:49.144055 14156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:49.146095 14156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "f4523d60820144cc924f3a533aae90b0"
format_stamp: "Formatted at 2026-06-20 07:16:49 on dist-test-slave-j663"
server_key: "e5f17ae3001c28824090891bd6864fed"
server_key_iv: "14a846b65bbcb1393e5e013f7b62fdfd"
server_key_version: "encryptionkey@0"
I20260620 07:16:49.146648 14156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "f4523d60820144cc924f3a533aae90b0"
format_stamp: "Formatted at 2026-06-20 07:16:49 on dist-test-slave-j663"
server_key: "e5f17ae3001c28824090891bd6864fed"
server_key_iv: "14a846b65bbcb1393e5e013f7b62fdfd"
server_key_version: "encryptionkey@0"
I20260620 07:16:49.150141 14156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 07:16:49.152433 14171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:49.153601 14156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:16:49.153718 14156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "f4523d60820144cc924f3a533aae90b0"
format_stamp: "Formatted at 2026-06-20 07:16:49 on dist-test-slave-j663"
server_key: "e5f17ae3001c28824090891bd6864fed"
server_key_iv: "14a846b65bbcb1393e5e013f7b62fdfd"
server_key_version: "encryptionkey@0"
I20260620 07:16:49.153817 14156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:49.170713 14156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:49.173540 14156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:49.173734 14156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:49.180855 14156 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:39325
I20260620 07:16:49.180879 14223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:39325 every 8 connection(s)
I20260620 07:16:49.181787 14156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:49.184388 14224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:49.185606 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 14156
I20260620 07:16:49.185695 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:16:49.185969 11758 external_mini_cluster.cc:1428] Setting key cfdb50c92a3602a86abaa331fcac65c7
I20260620 07:16:49.190891 14224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0: Bootstrap starting.
I20260620 07:16:49.193394 14224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:49.194103 14224 log.cc:824] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:16:49 dist-test-slave-j663 krb5kdc[14143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:49.196171 14224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0: No bootstrap required, opened a new log
I20260620 07:16:49.200429 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:49.187301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60098 (local address 127.11.123.190:39325)
0620 07:16:49.187708 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:49.187726 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:49.187769 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:16:49.188574 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:49.188597 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:49.189022 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:49.189395 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:49.190031 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.191398 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.192204 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.192855 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.193020 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:49.195435 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:49.195441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:49.195456 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:49.195485 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:49.197428 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:49.197817 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:49.197825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:49.197831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:49.197972 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:49.198217 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:49.198223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:49.198224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:49.198802 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:49.198999 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:49.199184 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 07:16:49.199473 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
I20260620 07:16:49.200729 14224 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4523d60820144cc924f3a533aae90b0" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39325 } }
I20260620 07:16:49.200961 14224 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:49.201026 14224 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4523d60820144cc924f3a533aae90b0, State: Initialized, Role: FOLLOWER
I20260620 07:16:49.201452 14224 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4523d60820144cc924f3a533aae90b0" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39325 } }
I20260620 07:16:49.201597 14224 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:49.201664 14224 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:49.201772 14224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:49.202685 14224 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4523d60820144cc924f3a533aae90b0" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39325 } }
I20260620 07:16:49.203008 14224 leader_election.cc:302] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4523d60820144cc924f3a533aae90b0; no voters:
I20260620 07:16:49.203300 14224 leader_election.cc:288] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:49.203404 14229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:49.203641 14229 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [term 1 LEADER]: Becoming Leader. State: Replica: f4523d60820144cc924f3a533aae90b0, State: Running, Role: LEADER
I20260620 07:16:49.203996 14229 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4523d60820144cc924f3a533aae90b0" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39325 } }
I20260620 07:16:49.204452 14224 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:49.205679 14231 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4523d60820144cc924f3a533aae90b0. Latest consensus state: current_term: 1 leader_uuid: "f4523d60820144cc924f3a533aae90b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4523d60820144cc924f3a533aae90b0" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39325 } } }
I20260620 07:16:49.205691 14230 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4523d60820144cc924f3a533aae90b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4523d60820144cc924f3a533aae90b0" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39325 } } }
I20260620 07:16:49.205807 14231 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:49.205813 14230 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:49.206162 14238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:49.209002 14238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:49.214391 14238 catalog_manager.cc:1347] Generated new cluster ID: 6800100d096e488593d83f2e3ccd4ab7
I20260620 07:16:49.214452 14238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:49.227617 14238 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:49.228508 14238 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:49.235989 14238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4523d60820144cc924f3a533aae90b0: Generated new TSK 0
I20260620 07:16:49.236660 14238 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:16:49.262203 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39325
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:49.369386 14250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:49.369633 14250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:49.369675 14250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:49.373529 14250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:49.373574 14250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:49.373627 14250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:16:49.378763 14250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:49.379675 14250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:49.380613 14250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:49.390066 14256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:49.390079 14258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:49.390231 14250 server_base.cc:1034] running on GCE node
W20260620 07:16:49.390111 14255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:49.390697 14250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:49.391340 14250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:49.392521 14250 hybrid_clock.cc:648] HybridClock initialized: now 1781939809392489 us; error 42 us; skew 500 ppm
Jun 20 07:16:49 dist-test-slave-j663 krb5kdc[14143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939809, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:49.395129 14250 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:16:49.395941 14250 webserver.cc:460] Webserver started at http://127.11.123.129:42895/ using document root <none> and password file <none>
I20260620 07:16:49.396430 14250 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:49.396486 14250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:49.396641 14250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:49.399056 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "53c9403dd83044818ce1d1785c0d0ddc"
format_stamp: "Formatted at 2026-06-20 07:16:49 on dist-test-slave-j663"
server_key: "f21dd0f65c6020b1464f3630ffa92459"
server_key_iv: "41e7c9d727e8aa8d911dfb95844548e4"
server_key_version: "encryptionkey@0"
I20260620 07:16:49.399534 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "53c9403dd83044818ce1d1785c0d0ddc"
format_stamp: "Formatted at 2026-06-20 07:16:49 on dist-test-slave-j663"
server_key: "f21dd0f65c6020b1464f3630ffa92459"
server_key_iv: "41e7c9d727e8aa8d911dfb95844548e4"
server_key_version: "encryptionkey@0"
I20260620 07:16:49.403009 14250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:16:49.405292 14265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:49.406474 14250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:16:49.406571 14250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "53c9403dd83044818ce1d1785c0d0ddc"
format_stamp: "Formatted at 2026-06-20 07:16:49 on dist-test-slave-j663"
server_key: "f21dd0f65c6020b1464f3630ffa92459"
server_key_iv: "41e7c9d727e8aa8d911dfb95844548e4"
server_key_version: "encryptionkey@0"
I20260620 07:16:49.406648 14250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:49.421757 14250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:49.424432 14250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:49.424579 14250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:49.425138 14250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:49.425925 14250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:49.425969 14250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:49.426030 14250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:49.426060 14250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:49.434810 14250 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:44801
I20260620 07:16:49.434839 14378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:44801 every 8 connection(s)
I20260620 07:16:49.435673 14250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:16:49.437855 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 14250
I20260620 07:16:49.437989 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:16:49.438263 11758 external_mini_cluster.cc:1428] Setting key d837fadc764a0a9b6c651c1ad5830e73
Jun 20 07:16:49 dist-test-slave-j663 krb5kdc[14143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939809, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:16:49.450069 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:49.437469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:34813 (local address 127.11.123.190:39325)
0620 07:16:49.437589 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:49.437593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:49.438745 (+ 1152us) server_negotiation.cc:407] Connection header received
0620 07:16:49.439817 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:49.439821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:49.439874 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:49.439961 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:49.441148 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.441738 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.443507 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.443809 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.443895 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:49.446417 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:49.446421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:49.446425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:49.446456 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:49.448076 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:49.448521 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:49.448524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:49.448525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:49.448597 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:49.449018 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:49.449021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:49.449022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:49.449268 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:49.449357 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:49.449848 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 07:16:49.449947 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
I20260620 07:16:49.450918 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:49.437874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:39325 (local address 127.11.123.129:34813)
0620 07:16:49.438585 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:49.438636 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:49.439593 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:49.440112 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:49.440119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:49.440462 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:49.440928 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:49.440964 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.441859 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.441864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:49.443387 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:49.443401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.443973 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.443979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:49.444185 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:49.444830 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:49.444846 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:49.446247 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:49.448185 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:49.448190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:49.448201 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:49.448435 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:49.448673 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:49.448675 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:49.448677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:49.448867 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:49.449362 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:49.449367 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:49.449588 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 07:16:49.449796 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 07:16:49.450063 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":131,"threads_started":1}
I20260620 07:16:49.452109 14379 heartbeater.cc:347] Connected to a master server at 127.11.123.190:39325
I20260620 07:16:49.452327 14379 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:49.452782 14379 heartbeater.cc:511] Master 127.11.123.190:39325 requested a full tablet report, sending...
I20260620 07:16:49.454244 14188 ts_manager.cc:194] Registered new tserver with Master: 53c9403dd83044818ce1d1785c0d0ddc (127.11.123.129:44801)
I20260620 07:16:49.455330 14188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:34813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:16:49.465381 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39325
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:49.575493 14384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:49.575731 14384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:49.575773 14384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:49.579828 14384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:49.579917 14384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:49.580013 14384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:16:49.585002 14384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:49.585958 14384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:49.587064 14384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:49.596758 14392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:49.596758 14389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:49.596844 14384 server_base.cc:1034] running on GCE node
W20260620 07:16:49.596776 14390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:49.597355 14384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:49.597998 14384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:49.599172 14384 hybrid_clock.cc:648] HybridClock initialized: now 1781939809599158 us; error 28 us; skew 500 ppm
Jun 20 07:16:49 dist-test-slave-j663 krb5kdc[14143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939809, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:49.601923 14384 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:16:49.602773 14384 webserver.cc:460] Webserver started at http://127.11.123.130:36061/ using document root <none> and password file <none>
I20260620 07:16:49.603312 14384 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:49.603406 14384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:49.603626 14384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:49.605825 14384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "917ab2e59b454a459e85822d21b9f8ec"
format_stamp: "Formatted at 2026-06-20 07:16:49 on dist-test-slave-j663"
server_key: "9d7f796d254bbc6bf5d90e369d4f73bb"
server_key_iv: "8609742b17d8b080ac9e259b6bed4b51"
server_key_version: "encryptionkey@0"
I20260620 07:16:49.606454 14384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "917ab2e59b454a459e85822d21b9f8ec"
format_stamp: "Formatted at 2026-06-20 07:16:49 on dist-test-slave-j663"
server_key: "9d7f796d254bbc6bf5d90e369d4f73bb"
server_key_iv: "8609742b17d8b080ac9e259b6bed4b51"
server_key_version: "encryptionkey@0"
I20260620 07:16:49.610740 14384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:16:49.613041 14399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:49.614279 14384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:16:49.614431 14384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "917ab2e59b454a459e85822d21b9f8ec"
format_stamp: "Formatted at 2026-06-20 07:16:49 on dist-test-slave-j663"
server_key: "9d7f796d254bbc6bf5d90e369d4f73bb"
server_key_iv: "8609742b17d8b080ac9e259b6bed4b51"
server_key_version: "encryptionkey@0"
I20260620 07:16:49.614538 14384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:49.629578 14384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:49.632550 14384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:49.632766 14384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:49.633419 14384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:49.634225 14384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:49.634291 14384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:49.634358 14384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:49.634407 14384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:49.643769 14384 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:41149
I20260620 07:16:49.643786 14512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:41149 every 8 connection(s)
I20260620 07:16:49.644675 14384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:16:49.651185 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 14384
I20260620 07:16:49.651314 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:16:49.651607 11758 external_mini_cluster.cc:1428] Setting key b75553470f619641dff3241cb7655991
Jun 20 07:16:49 dist-test-slave-j663 krb5kdc[14143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939809, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:49.659473 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:49.646535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:38551 (local address 127.11.123.190:39325)
0620 07:16:49.646684 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:49.646689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:49.647753 (+ 1064us) server_negotiation.cc:407] Connection header received
0620 07:16:49.648608 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:49.648611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:49.648658 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:49.648737 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:49.650097 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.650654 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.652277 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.652566 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.652689 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:49.655499 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:49.655502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:49.655510 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:49.655535 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:49.657022 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:49.657796 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:49.657799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:49.657800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:49.657881 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:49.658330 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:49.658332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:49.658333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:49.658541 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:49.658649 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:49.659171 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 07:16:49.659293 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":56}
I20260620 07:16:49.660346 14515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:49.646969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:39325 (local address 127.11.123.130:38551)
0620 07:16:49.647606 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:49.647645 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:49.648420 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:49.648934 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:49.648942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:49.649434 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:49.649934 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:49.649949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.650782 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.650785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:49.652119 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:49.652129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.652731 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.652735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:49.652956 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:49.653676 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:49.653694 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:49.655316 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:49.657195 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:49.657205 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:49.657226 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:49.657676 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:49.657976 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:49.657982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:49.657986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:49.658230 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:49.658649 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:49.658657 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:49.658932 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 07:16:49.659132 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 07:16:49.659469 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
I20260620 07:16:49.661537 14513 heartbeater.cc:347] Connected to a master server at 127.11.123.190:39325
I20260620 07:16:49.661785 14513 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:49.662271 14513 heartbeater.cc:511] Master 127.11.123.190:39325 requested a full tablet report, sending...
I20260620 07:16:49.663306 14188 ts_manager.cc:194] Registered new tserver with Master: 917ab2e59b454a459e85822d21b9f8ec (127.11.123.130:41149)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
I20260620 07:16:49.663897 14188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:38551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:16:49.678501 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39325
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:49.787552 14518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:49.787804 14518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:49.787846 14518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:49.791848 14518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:49.791908 14518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:49.791965 14518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:16:49.797317 14518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:49.798277 14518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:49.799233 14518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:49.809288 14523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:49.809309 14524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:49.809358 14526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:49.809633 14518 server_base.cc:1034] running on GCE node
I20260620 07:16:49.810076 14518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:49.810708 14518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:49.811895 14518 hybrid_clock.cc:648] HybridClock initialized: now 1781939809811872 us; error 43 us; skew 500 ppm
Jun 20 07:16:49 dist-test-slave-j663 krb5kdc[14143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939809, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:49.814695 14518 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:16:49.815541 14518 webserver.cc:460] Webserver started at http://127.11.123.131:46807/ using document root <none> and password file <none>
I20260620 07:16:49.816102 14518 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:49.816187 14518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:49.816408 14518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:49.818622 14518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "f7efad6a63dd44a7a2d82b8145e3175f"
format_stamp: "Formatted at 2026-06-20 07:16:49 on dist-test-slave-j663"
server_key: "abf43f14534c594b32f6672347ef904d"
server_key_iv: "a3c067c95472baf84514fc2604a268a5"
server_key_version: "encryptionkey@0"
I20260620 07:16:49.819190 14518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "f7efad6a63dd44a7a2d82b8145e3175f"
format_stamp: "Formatted at 2026-06-20 07:16:49 on dist-test-slave-j663"
server_key: "abf43f14534c594b32f6672347ef904d"
server_key_iv: "a3c067c95472baf84514fc2604a268a5"
server_key_version: "encryptionkey@0"
I20260620 07:16:49.825115 14518 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 07:16:49.828320 14533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:49.829720 14518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:16:49.829854 14518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "f7efad6a63dd44a7a2d82b8145e3175f"
format_stamp: "Formatted at 2026-06-20 07:16:49 on dist-test-slave-j663"
server_key: "abf43f14534c594b32f6672347ef904d"
server_key_iv: "a3c067c95472baf84514fc2604a268a5"
server_key_version: "encryptionkey@0"
I20260620 07:16:49.829964 14518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:49.848531 14518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:49.851795 14518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:49.851992 14518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:49.852587 14518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:49.853451 14518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:49.853516 14518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:49.853582 14518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:49.853627 14518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:49.862636 14518 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:40759
I20260620 07:16:49.862670 14646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:40759 every 8 connection(s)
I20260620 07:16:49.863545 14518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:16:49.864355 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 14518
I20260620 07:16:49.864475 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:16:49.864737 11758 external_mini_cluster.cc:1428] Setting key 81de153e7966736118dc4d096dc5ba67
Jun 20 07:16:49 dist-test-slave-j663 krb5kdc[14143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939809, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:49.877137 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:49.865484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:39063 (local address 127.11.123.190:39325)
0620 07:16:49.865644 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:49.865649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:49.866899 (+ 1250us) server_negotiation.cc:407] Connection header received
0620 07:16:49.867809 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:49.867812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:49.867863 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:49.867939 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:49.869037 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.869617 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.870736 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.870982 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.871093 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:49.873568 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:49.873571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:49.873574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:49.873600 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:49.874934 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:49.875541 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:49.875545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:49.875547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:49.875633 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:49.875986 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:49.875991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:49.875992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:49.876183 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:49.876304 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:49.876804 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 07:16:49.876938 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:16:49.878132 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:49.865935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:39325 (local address 127.11.123.131:39063)
0620 07:16:49.866756 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:49.866811 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:49.867615 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:49.868102 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:49.868109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:49.868451 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:49.868903 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:49.868917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.869742 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.869747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:49.870612 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:49.870621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:49.871125 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:49.871129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:49.871306 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:49.871832 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:49.871848 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:49.873395 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:49.875159 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:49.875164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:49.875179 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:49.875431 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:49.875744 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:49.875747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:49.875749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:49.875887 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:49.876312 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:49.876320 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:49.876558 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 07:16:49.876764 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 07:16:49.877121 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":141,"threads_started":1}
I20260620 07:16:49.879498 14647 heartbeater.cc:347] Connected to a master server at 127.11.123.190:39325
I20260620 07:16:49.879736 14647 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:49.880173 14647 heartbeater.cc:511] Master 127.11.123.190:39325 requested a full tablet report, sending...
I20260620 07:16:49.881234 14187 ts_manager.cc:194] Registered new tserver with Master: f7efad6a63dd44a7a2d82b8145e3175f (127.11.123.131:40759)
I20260620 07:16:49.881832 14187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:39063
I20260620 07:16:49.890831 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 07:16:49.903258 14227 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:49.901978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.11.123.190:39325)
0620 07:16:49.902144 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:49.902148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:49.902235 (+ 87us) server_negotiation.cc:407] Connection header received
0620 07:16:49.902308 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:49.902311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:49.902370 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:49.902467 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:49.903135 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60106: BlockingRecv error: recv got EOF from 127.0.0.1:60106 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:16:49.904310 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 14250
I20260620 07:16:49.909930 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 14384
I20260620 07:16:49.915117 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 14518
I20260620 07:16:49.920334 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 14156
2026-06-20T07:16:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2105 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:49 dist-test-slave-j663 krb5kdc[14659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:49 dist-test-slave-j663 krb5kdc[14659](info): set up 2 sockets
Jun 20 07:16:49 dist-test-slave-j663 krb5kdc[14659](info): commencing operation
krb5kdc: starting...
W20260620 07:16:50.968989 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:16:51 dist-test-slave-j663 krb5kdc[14659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:16:51.088352 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:43961
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:43961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:51.194762 14672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:51.195024 14672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:51.195065 14672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:51.198963 14672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:51.199012 14672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:51.199031 14672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:51.199046 14672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:51.199061 14672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:51.204016 14672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:43961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:43961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:51.204962 14672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:51.205906 14672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:51.214267 14680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:51.214275 14678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:51.214282 14677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:51.214475 14672 server_base.cc:1034] running on GCE node
I20260620 07:16:51.215122 14672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:51.216255 14672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:51.217432 14672 hybrid_clock.cc:648] HybridClock initialized: now 1781939811217400 us; error 47 us; skew 500 ppm
Jun 20 07:16:51 dist-test-slave-j663 krb5kdc[14659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939811, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:51.220124 14672 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:16:51.221061 14672 webserver.cc:460] Webserver started at http://127.11.123.190:36013/ using document root <none> and password file <none>
I20260620 07:16:51.221659 14672 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:51.221747 14672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:51.221979 14672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:51.223989 14672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "7460c37bf71d42349450f61bc664140d"
format_stamp: "Formatted at 2026-06-20 07:16:51 on dist-test-slave-j663"
server_key: "1f380323ace8b861a213ec14de4fab3f"
server_key_iv: "3ea4a6780a499ab25971382c640796a0"
server_key_version: "encryptionkey@0"
I20260620 07:16:51.224558 14672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "7460c37bf71d42349450f61bc664140d"
format_stamp: "Formatted at 2026-06-20 07:16:51 on dist-test-slave-j663"
server_key: "1f380323ace8b861a213ec14de4fab3f"
server_key_iv: "3ea4a6780a499ab25971382c640796a0"
server_key_version: "encryptionkey@0"
I20260620 07:16:51.228219 14672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 07:16:51.230583 14687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:51.231779 14672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:16:51.231922 14672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "7460c37bf71d42349450f61bc664140d"
format_stamp: "Formatted at 2026-06-20 07:16:51 on dist-test-slave-j663"
server_key: "1f380323ace8b861a213ec14de4fab3f"
server_key_iv: "3ea4a6780a499ab25971382c640796a0"
server_key_version: "encryptionkey@0"
I20260620 07:16:51.232028 14672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:51.250483 14672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:51.253593 14672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:51.253793 14672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:51.260727 14672 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:43961
I20260620 07:16:51.261034 14739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:43961 every 8 connection(s)
I20260620 07:16:51.261745 14672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:51.263940 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 14672
I20260620 07:16:51.264068 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:16:51.264386 11758 external_mini_cluster.cc:1428] Setting key 3512290986c2924b8839c63ef4658115
I20260620 07:16:51.264688 14740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:51.271508 14740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d: Bootstrap starting.
Jun 20 07:16:51 dist-test-slave-j663 krb5kdc[14659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:51.274130 14740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:51.274820 14740 log.cc:824] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:51.276551 14740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d: No bootstrap required, opened a new log
I20260620 07:16:51.278225 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:51.265687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.11.123.190:43961)
0620 07:16:51.266081 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:51.266096 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:51.266130 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:16:51.266756 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:51.266770 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:51.267090 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:51.267366 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:51.268297 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:51.269329 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:51.270519 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:51.270961 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:51.271071 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:51.273864 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:51.273869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:51.273881 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:51.273907 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:51.275568 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:51.276003 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:51.276007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:51.276012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:51.276112 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:51.276396 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:51.276399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:51.276401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:51.276836 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:51.277024 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:51.277296 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 07:16:51.277564 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":137,"threads_started":1}
I20260620 07:16:51.280396 14740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7460c37bf71d42349450f61bc664140d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43961 } }
I20260620 07:16:51.280673 14740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:51.280735 14740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7460c37bf71d42349450f61bc664140d, State: Initialized, Role: FOLLOWER
I20260620 07:16:51.281164 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:16:51.281209 14740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7460c37bf71d42349450f61bc664140d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43961 } }
I20260620 07:16:51.281378 14740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:51.281440 14740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:51.281530 14740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:51.282435 14740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7460c37bf71d42349450f61bc664140d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43961 } }
I20260620 07:16:51.282764 14740 leader_election.cc:302] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7460c37bf71d42349450f61bc664140d; no voters:
I20260620 07:16:51.283094 14740 leader_election.cc:288] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:51.283166 14746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:51.283392 14746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [term 1 LEADER]: Becoming Leader. State: Replica: 7460c37bf71d42349450f61bc664140d, State: Running, Role: LEADER
I20260620 07:16:51.283723 14746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7460c37bf71d42349450f61bc664140d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43961 } }
I20260620 07:16:51.284303 14740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:51.284848 14746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7460c37bf71d42349450f61bc664140d. Latest consensus state: current_term: 1 leader_uuid: "7460c37bf71d42349450f61bc664140d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7460c37bf71d42349450f61bc664140d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43961 } } }
I20260620 07:16:51.284958 14746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:51.285280 14747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7460c37bf71d42349450f61bc664140d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7460c37bf71d42349450f61bc664140d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43961 } } }
I20260620 07:16:51.285390 14750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:51.285463 14747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:51.289494 14750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:51.296546 14750 catalog_manager.cc:1347] Generated new cluster ID: b5e32abdcf514e30b3c9c8fce9c56577
I20260620 07:16:51.296612 14750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:51.299887 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:51.294489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.11.123.190:43961)
0620 07:16:51.294621 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:51.294626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:51.294823 (+ 197us) server_negotiation.cc:407] Connection header received
0620 07:16:51.294951 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:51.294954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:51.295000 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:51.295079 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:51.295658 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:51.296419 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:51.297638 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:51.297986 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:51.298076 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:51.298968 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:51.298972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:51.298975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:51.299454 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:51.299483 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:51.299500 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:51.299583 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:51.299709 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 07:16:51.299757 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:16:51.309270 14750 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:51.310233 14750 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:51.315212 14750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7460c37bf71d42349450f61bc664140d: Generated new TSK 0
I20260620 07:16:51.315979 14750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:16:51.325650 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 14672
2026-06-20T07:16:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1403 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:51 dist-test-slave-j663 krb5kdc[14773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:51 dist-test-slave-j663 krb5kdc[14773](info): set up 2 sockets
Jun 20 07:16:51 dist-test-slave-j663 krb5kdc[14773](info): commencing operation
krb5kdc: starting...
W20260620 07:16:52.369184 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:16:52 dist-test-slave-j663 krb5kdc[14773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:16:52.487547 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:43893
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:43893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:52.594558 14786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:52.594812 14786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:52.594853 14786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:52.598779 14786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:52.598827 14786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:52.598847 14786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:52.598863 14786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:52.598878 14786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:52.603966 14786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:43893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:43893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:52.604926 14786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:52.605888 14786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:52.614241 14791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:52.614259 14792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:52.614241 14786 server_base.cc:1034] running on GCE node
W20260620 07:16:52.614255 14794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:52.615039 14786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:52.616106 14786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:52.617300 14786 hybrid_clock.cc:648] HybridClock initialized: now 1781939812617276 us; error 40 us; skew 500 ppm
Jun 20 07:16:52 dist-test-slave-j663 krb5kdc[14773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939812, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:52.620078 14786 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:16:52.621006 14786 webserver.cc:460] Webserver started at http://127.11.123.190:45611/ using document root <none> and password file <none>
I20260620 07:16:52.621587 14786 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:52.621675 14786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:52.621902 14786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:52.623916 14786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "47cfd0405b4c4b4a9cd9b27f583198cd"
format_stamp: "Formatted at 2026-06-20 07:16:52 on dist-test-slave-j663"
server_key: "69efe9abc47d37d87e11852c75b38e86"
server_key_iv: "60ca8b52d72108ee1c4aaef1c7042dff"
server_key_version: "encryptionkey@0"
I20260620 07:16:52.624485 14786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "47cfd0405b4c4b4a9cd9b27f583198cd"
format_stamp: "Formatted at 2026-06-20 07:16:52 on dist-test-slave-j663"
server_key: "69efe9abc47d37d87e11852c75b38e86"
server_key_iv: "60ca8b52d72108ee1c4aaef1c7042dff"
server_key_version: "encryptionkey@0"
I20260620 07:16:52.628067 14786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:16:52.630432 14801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:52.631639 14786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:16:52.631772 14786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "47cfd0405b4c4b4a9cd9b27f583198cd"
format_stamp: "Formatted at 2026-06-20 07:16:52 on dist-test-slave-j663"
server_key: "69efe9abc47d37d87e11852c75b38e86"
server_key_iv: "60ca8b52d72108ee1c4aaef1c7042dff"
server_key_version: "encryptionkey@0"
I20260620 07:16:52.631880 14786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:52.649183 14786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:52.652412 14786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:52.652611 14786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:52.659475 14786 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:43893
I20260620 07:16:52.659489 14853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:43893 every 8 connection(s)
I20260620 07:16:52.660441 14786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:52.662865 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 14786
I20260620 07:16:52.662995 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:16:52.663283 11758 external_mini_cluster.cc:1428] Setting key 43c5c381ee571df2543baf065f99a4ac
I20260620 07:16:52.663412 14854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:52.670254 14854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd: Bootstrap starting.
Jun 20 07:16:52 dist-test-slave-j663 krb5kdc[14773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:52.672883 14854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:52.673648 14854 log.cc:824] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:52.675879 14854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd: No bootstrap required, opened a new log
I20260620 07:16:52.678470 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:52.664692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.11.123.190:43893)
0620 07:16:52.665111 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:52.665132 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:52.665172 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:16:52.666126 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:52.666143 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:52.666478 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:52.666750 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:52.667630 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:52.668588 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:52.669728 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:52.670172 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:52.670288 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:52.673010 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:52.673018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:52.673039 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:52.673067 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:52.675218 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:52.675627 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:52.675632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:52.675640 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:52.675803 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:52.676083 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:52.676086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:52.676089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:52.676610 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:52.676800 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:52.677097 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 07:16:52.677425 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":139,"threads_started":1}
I20260620 07:16:52.680290 14854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cfd0405b4c4b4a9cd9b27f583198cd" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43893 } }
I20260620 07:16:52.680550 14854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:52.680603 14854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cfd0405b4c4b4a9cd9b27f583198cd, State: Initialized, Role: FOLLOWER
I20260620 07:16:52.681123 14854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cfd0405b4c4b4a9cd9b27f583198cd" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43893 } }
I20260620 07:16:52.681288 14854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:52.681353 14854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:52.681448 14854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:52.682337 14854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cfd0405b4c4b4a9cd9b27f583198cd" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43893 } }
I20260620 07:16:52.682667 14854 leader_election.cc:302] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47cfd0405b4c4b4a9cd9b27f583198cd; no voters:
I20260620 07:16:52.682989 14854 leader_election.cc:288] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:52.683130 14859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:52.683398 14859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [term 1 LEADER]: Becoming Leader. State: Replica: 47cfd0405b4c4b4a9cd9b27f583198cd, State: Running, Role: LEADER
I20260620 07:16:52.683740 14859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cfd0405b4c4b4a9cd9b27f583198cd" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43893 } }
I20260620 07:16:52.684203 14854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:52.685529 14860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47cfd0405b4c4b4a9cd9b27f583198cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cfd0405b4c4b4a9cd9b27f583198cd" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43893 } } }
I20260620 07:16:52.685527 14861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47cfd0405b4c4b4a9cd9b27f583198cd. Latest consensus state: current_term: 1 leader_uuid: "47cfd0405b4c4b4a9cd9b27f583198cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cfd0405b4c4b4a9cd9b27f583198cd" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43893 } } }
I20260620 07:16:52.685761 14861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:52.685762 14860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd [sys.catalog]: This master's current role is: LEADER
W20260620 07:16:52.688665 14874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:16:52.688736 14874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:16:52.688817 14870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:52.689556 14870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:52.694299 14870 catalog_manager.cc:1347] Generated new cluster ID: d61e9f91b603406292232b075a0d85e6
I20260620 07:16:52.694358 14870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:52.713892 14870 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:52.715076 14870 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:52.729722 14870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47cfd0405b4c4b4a9cd9b27f583198cd: Generated new TSK 0
I20260620 07:16:52.730500 14870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:16:52.760548 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:43893
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:52.867769 14880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:52.868047 14880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:52.868124 14880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:52.872035 14880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:52.872118 14880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:52.872206 14880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:16:52.877259 14880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:43893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:52.878250 14880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:52.879251 14880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:52.888741 14886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:52.888705 14888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:52.888753 14880 server_base.cc:1034] running on GCE node
W20260620 07:16:52.888705 14885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:52.889446 14880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:52.890156 14880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:52.891343 14880 hybrid_clock.cc:648] HybridClock initialized: now 1781939812891296 us; error 56 us; skew 500 ppm
Jun 20 07:16:52 dist-test-slave-j663 krb5kdc[14773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939812, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:52.900285 14880 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:16:52.901244 14880 webserver.cc:460] Webserver started at http://127.11.123.129:46031/ using document root <none> and password file <none>
I20260620 07:16:52.901810 14880 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:52.901914 14880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:52.902145 14880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:52.904171 14880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "02b44140b8834f0caf8eb1e26a51dc91"
format_stamp: "Formatted at 2026-06-20 07:16:52 on dist-test-slave-j663"
server_key: "1abdf8e448461f1f4ba9c951f54b159a"
server_key_iv: "fffbdccb8fcc189cb2c0a482360160ee"
server_key_version: "encryptionkey@0"
I20260620 07:16:52.904749 14880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "02b44140b8834f0caf8eb1e26a51dc91"
format_stamp: "Formatted at 2026-06-20 07:16:52 on dist-test-slave-j663"
server_key: "1abdf8e448461f1f4ba9c951f54b159a"
server_key_iv: "fffbdccb8fcc189cb2c0a482360160ee"
server_key_version: "encryptionkey@0"
I20260620 07:16:52.908383 14880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:16:52.910789 14895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:52.911998 14880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:16:52.912132 14880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "02b44140b8834f0caf8eb1e26a51dc91"
format_stamp: "Formatted at 2026-06-20 07:16:52 on dist-test-slave-j663"
server_key: "1abdf8e448461f1f4ba9c951f54b159a"
server_key_iv: "fffbdccb8fcc189cb2c0a482360160ee"
server_key_version: "encryptionkey@0"
I20260620 07:16:52.912241 14880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:52.932320 14880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:52.935689 14880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:52.935925 14880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:52.936503 14880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:52.937330 14880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:52.937379 14880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:52.937418 14880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:52.937440 14880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:52.945927 14880 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:32959
I20260620 07:16:52.945950 15008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:32959 every 8 connection(s)
I20260620 07:16:52.946768 14880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
Jun 20 07:16:52 dist-test-slave-j663 krb5kdc[14773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939812, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:52.956594 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 14880
I20260620 07:16:52.956730 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:16:52.956993 11758 external_mini_cluster.cc:1428] Setting key 3097d2ce626c35356183e37bdf613fb0
I20260620 07:16:52.959939 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:52.948517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:43653 (local address 127.11.123.190:43893)
0620 07:16:52.948646 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:52.948649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:52.949515 (+ 866us) server_negotiation.cc:407] Connection header received
0620 07:16:52.950869 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:52.950872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:52.950917 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:52.950992 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:52.952073 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:52.952589 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:52.953735 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:52.953960 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:52.954040 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:52.956243 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:52.956246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:52.956248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:52.956269 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:52.957848 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:52.958412 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:52.958416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:52.958417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:52.958502 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:52.958887 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:52.958890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:52.958891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:52.959063 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:52.959141 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:52.959720 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 07:16:52.959817 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:16:52.960865 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:52.948888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:43893 (local address 127.11.123.129:43653)
0620 07:16:52.949374 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:52.949416 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:52.950677 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:52.951152 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:52.951158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:52.951509 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:52.951954 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:52.951966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:52.952692 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:52.952694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:52.953614 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:52.953622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:52.954082 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:52.954084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:52.954224 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:52.954744 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:52.954759 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:52.956099 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:52.957950 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:52.957955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:52.957968 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:52.958268 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:52.958621 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:52.958624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:52.958626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:52.958790 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:52.959171 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:52.959180 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:52.959483 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 07:16:52.959687 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 07:16:52.960021 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":157,"threads_started":1}
I20260620 07:16:52.962026 15009 heartbeater.cc:347] Connected to a master server at 127.11.123.190:43893
I20260620 07:16:52.962266 15009 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:52.962844 15009 heartbeater.cc:511] Master 127.11.123.190:43893 requested a full tablet report, sending...
I20260620 07:16:52.964570 14818 ts_manager.cc:194] Registered new tserver with Master: 02b44140b8834f0caf8eb1e26a51dc91 (127.11.123.129:32959)
I20260620 07:16:52.966260 14818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:43653
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:16:52.983361 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:43893
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:53.093431 15014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:53.093746 15014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:53.093834 15014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:53.098034 15014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:53.098130 15014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:53.098224 15014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:16:53.103605 15014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:43893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:53.104779 15014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:53.105867 15014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:53.115687 15022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:53.115718 15020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:53.115720 15019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:53.116096 15014 server_base.cc:1034] running on GCE node
I20260620 07:16:53.116528 15014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:53.117252 15014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:53.118438 15014 hybrid_clock.cc:648] HybridClock initialized: now 1781939813118410 us; error 50 us; skew 500 ppm
Jun 20 07:16:53 dist-test-slave-j663 krb5kdc[14773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939813, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:53.121162 15014 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:16:53.122071 15014 webserver.cc:460] Webserver started at http://127.11.123.130:41825/ using document root <none> and password file <none>
I20260620 07:16:53.122665 15014 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:53.122757 15014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:53.122969 15014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:53.124972 15014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "5691aee35371449c8acb37d8f63dda07"
format_stamp: "Formatted at 2026-06-20 07:16:53 on dist-test-slave-j663"
server_key: "eb406d05a3a1c70637cac85b12f8177d"
server_key_iv: "5ec338b64b67c6842778c1a4aa282ecb"
server_key_version: "encryptionkey@0"
I20260620 07:16:53.125587 15014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "5691aee35371449c8acb37d8f63dda07"
format_stamp: "Formatted at 2026-06-20 07:16:53 on dist-test-slave-j663"
server_key: "eb406d05a3a1c70637cac85b12f8177d"
server_key_iv: "5ec338b64b67c6842778c1a4aa282ecb"
server_key_version: "encryptionkey@0"
I20260620 07:16:53.129177 15014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:16:53.131501 15029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:53.132853 15014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:53.133005 15014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "5691aee35371449c8acb37d8f63dda07"
format_stamp: "Formatted at 2026-06-20 07:16:53 on dist-test-slave-j663"
server_key: "eb406d05a3a1c70637cac85b12f8177d"
server_key_iv: "5ec338b64b67c6842778c1a4aa282ecb"
server_key_version: "encryptionkey@0"
I20260620 07:16:53.133235 15014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:53.147876 15014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:53.150933 15014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:53.151161 15014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:53.151793 15014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:53.152675 15014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:53.152750 15014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:53.152827 15014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:53.152875 15014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:53.162580 15014 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:32809
I20260620 07:16:53.162606 15142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:32809 every 8 connection(s)
I20260620 07:16:53.163502 15014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:16:53.169442 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 15014
I20260620 07:16:53.169562 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:16:53.169852 11758 external_mini_cluster.cc:1428] Setting key c16a472f898bed2c1de0e27138d23d57
Jun 20 07:16:53 dist-test-slave-j663 krb5kdc[14773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939813, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:53.178613 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:53.165607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:34801 (local address 127.11.123.190:43893)
0620 07:16:53.165739 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:53.165743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:53.166531 (+ 788us) server_negotiation.cc:407] Connection header received
0620 07:16:53.167885 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:53.167889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:53.167946 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:53.168051 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:53.169205 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:53.170002 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:53.171674 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:53.172044 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:53.172167 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:53.174787 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:53.174790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:53.174791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:53.174814 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:53.176288 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:53.176930 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:53.176932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:53.176933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:53.177015 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:53.177441 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:53.177444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:53.177446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:53.177698 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:53.177790 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:53.178404 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 07:16:53.178498 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
I20260620 07:16:53.179811 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:53.165931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:43893 (local address 127.11.123.130:34801)
0620 07:16:53.166378 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:53.166419 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:53.167670 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:53.168223 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:53.168229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:53.168574 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:53.169039 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:53.169052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:53.170155 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:53.170158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:53.171554 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:53.171563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:53.172208 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:53.172211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:53.172359 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:53.173026 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:53.173041 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:53.174629 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:53.176414 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:53.176422 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:53.176441 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:53.176818 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:53.177136 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:53.177142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:53.177147 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:53.177322 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:53.177825 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:53.177832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:53.178142 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 07:16:53.178366 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 07:16:53.178699 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260620 07:16:53.181015 15143 heartbeater.cc:347] Connected to a master server at 127.11.123.190:43893
I20260620 07:16:53.181318 15143 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:53.181807 15143 heartbeater.cc:511] Master 127.11.123.190:43893 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
I20260620 07:16:53.182878 14818 ts_manager.cc:194] Registered new tserver with Master: 5691aee35371449c8acb37d8f63dda07 (127.11.123.130:32809)
I20260620 07:16:53.183665 14818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:34801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:16:53.197983 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:43893
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:53.317143 15148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:53.317422 15148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:53.317471 15148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:53.321651 15148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:53.321708 15148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:53.321766 15148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:16:53.326851 15148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:43893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:53.327806 15148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:53.328744 15148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:53.338124 15156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:53.338212 15154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:53.338158 15153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:53.338395 15148 server_base.cc:1034] running on GCE node
I20260620 07:16:53.338783 15148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:53.339432 15148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:53.340628 15148 hybrid_clock.cc:648] HybridClock initialized: now 1781939813340613 us; error 37 us; skew 500 ppm
Jun 20 07:16:53 dist-test-slave-j663 krb5kdc[14773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939813, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:53.343530 15148 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:16:53.344378 15148 webserver.cc:460] Webserver started at http://127.11.123.131:36517/ using document root <none> and password file <none>
I20260620 07:16:53.344925 15148 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:53.344983 15148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:53.345249 15148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:53.347268 15148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "03ea6372e49446d68b234dd741c42ba4"
format_stamp: "Formatted at 2026-06-20 07:16:53 on dist-test-slave-j663"
server_key: "d0719bc44f400f6af0082e1a3384b353"
server_key_iv: "edf36cf04518ac17f4ad88169d5e170e"
server_key_version: "encryptionkey@0"
I20260620 07:16:53.347764 15148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "03ea6372e49446d68b234dd741c42ba4"
format_stamp: "Formatted at 2026-06-20 07:16:53 on dist-test-slave-j663"
server_key: "d0719bc44f400f6af0082e1a3384b353"
server_key_iv: "edf36cf04518ac17f4ad88169d5e170e"
server_key_version: "encryptionkey@0"
I20260620 07:16:53.351419 15148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:16:53.353852 15163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:53.355062 15148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:16:53.355185 15148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "03ea6372e49446d68b234dd741c42ba4"
format_stamp: "Formatted at 2026-06-20 07:16:53 on dist-test-slave-j663"
server_key: "d0719bc44f400f6af0082e1a3384b353"
server_key_iv: "edf36cf04518ac17f4ad88169d5e170e"
server_key_version: "encryptionkey@0"
I20260620 07:16:53.355290 15148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:53.370025 15148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:53.372678 15148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:53.372871 15148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:53.373482 15148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:53.374258 15148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:53.374302 15148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:53.374366 15148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:53.374399 15148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:53.383016 15148 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:42537
I20260620 07:16:53.383037 15276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:42537 every 8 connection(s)
I20260620 07:16:53.383886 15148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
Jun 20 07:16:53 dist-test-slave-j663 krb5kdc[14773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939813, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:53.394068 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 15148
I20260620 07:16:53.394191 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:16:53.394488 11758 external_mini_cluster.cc:1428] Setting key fa5bb1ee656a2540da22043019ae9979
I20260620 07:16:53.397608 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:53.385854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:47687 (local address 127.11.123.190:43893)
0620 07:16:53.385984 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:53.385988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:53.386865 (+ 877us) server_negotiation.cc:407] Connection header received
0620 07:16:53.388199 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:53.388203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:53.388261 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:53.388374 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:53.389567 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:53.390100 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:53.391223 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:53.391453 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:53.391568 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:53.394026 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:53.394030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:53.394032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:53.394065 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:53.395681 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:53.396187 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:53.396191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:53.396193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:53.396280 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:53.396652 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:53.396654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:53.396655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:53.396823 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:53.396898 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:53.397406 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 07:16:53.397496 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:16:53.398391 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:53.386264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:43893 (local address 127.11.123.131:47687)
0620 07:16:53.386717 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:53.386758 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:53.387991 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:53.388538 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:53.388545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:53.388897 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:53.389423 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:53.389436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:53.390226 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:53.390230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:53.391098 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:53.391106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:53.391602 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:53.391606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:53.391773 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:53.392372 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:53.392389 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:53.393864 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:53.395768 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:53.395774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:53.395787 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:53.396091 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:53.396395 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:53.396401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:53.396406 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:53.396562 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:53.396909 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:53.396914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:53.397181 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 07:16:53.397340 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 07:16:53.397579 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
I20260620 07:16:53.399494 15277 heartbeater.cc:347] Connected to a master server at 127.11.123.190:43893
I20260620 07:16:53.399727 15277 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:53.400203 15277 heartbeater.cc:511] Master 127.11.123.190:43893 requested a full tablet report, sending...
I20260620 07:16:53.401206 14818 ts_manager.cc:194] Registered new tserver with Master: 03ea6372e49446d68b234dd741c42ba4 (127.11.123.131:42537)
I20260620 07:16:53.401748 14818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:47687
I20260620 07:16:53.408304 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:16:53 dist-test-slave-j663 krb5kdc[14773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939813, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 07:16:53 dist-test-slave-j663 krb5kdc[14773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939813, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:53.434149 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:53.425388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.11.123.190:43893)
0620 07:16:53.425528 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:53.425535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:53.425713 (+ 178us) server_negotiation.cc:407] Connection header received
0620 07:16:53.425902 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:53.425905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:53.425980 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:53.426093 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:53.426947 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:53.427851 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:53.428678 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:53.428912 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:53.428996 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:53.431457 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:53.431459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:53.431461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:53.431484 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:53.432695 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:53.433063 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:53.433065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:53.433067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:53.433205 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:53.433438 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:53.433440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:53.433442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:53.433641 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:53.433741 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:53.433941 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 07:16:53.434033 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 07:16:53.434556 14818 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36832
I20260620 07:16:53.435827 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 14880
I20260620 07:16:53.441009 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 15014
I20260620 07:16:53.446290 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 15148
I20260620 07:16:53.451300 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 14786
2026-06-20T07:16:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2127 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:53 dist-test-slave-j663 krb5kdc[15289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:53 dist-test-slave-j663 krb5kdc[15289](info): set up 2 sockets
Jun 20 07:16:53 dist-test-slave-j663 krb5kdc[15289](info): commencing operation
krb5kdc: starting...
W20260620 07:16:54.593951 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 07:16:54.832599 11758 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.238s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 07:16:55.006301 11758 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.174s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 07:16:55.309011 11758 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.303s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/test-user.keytab.
W20260620 07:16:56.101151 11758 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.792s user 0.002s sys 0.000s
Jun 20 07:16:56 dist-test-slave-j663 krb5kdc[15289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
W20260620 07:16:57.427862 11758 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.11.123.190: real 1.305s user 0.001s sys 0.002s
I20260620 07:16:57.428450 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:34881
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:34881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:57.547171 15302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:57.547434 15302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:57.547475 15302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:57.551359 15302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:57.551414 15302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:57.551432 15302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:57.551450 15302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:57.551465 15302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:57.556634 15302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:34881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:34881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:57.557704 15302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:57.558727 15302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:57.567345 15307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:57.567514 15302 server_base.cc:1034] running on GCE node
W20260620 07:16:57.567344 15310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:57.567354 15308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:57.568248 15302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:57.569502 15302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:57.570704 15302 hybrid_clock.cc:648] HybridClock initialized: now 1781939817570680 us; error 32 us; skew 500 ppm
Jun 20 07:16:57 dist-test-slave-j663 krb5kdc[15289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939817, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:57.573716 15302 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:16:57.574889 15302 webserver.cc:460] Webserver started at http://127.11.123.190:39973/ using document root <none> and password file <none>
I20260620 07:16:57.575555 15302 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:57.575651 15302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:57.575956 15302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:57.578181 15302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "aeb76548e48841cfb56807443fac2e43"
format_stamp: "Formatted at 2026-06-20 07:16:57 on dist-test-slave-j663"
server_key: "844e8589b1654fd2a6df2943a296ef9d"
server_key_iv: "1ad1fe8caeca636d034fcbbc994e5367"
server_key_version: "encryptionkey@0"
I20260620 07:16:57.578758 15302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "aeb76548e48841cfb56807443fac2e43"
format_stamp: "Formatted at 2026-06-20 07:16:57 on dist-test-slave-j663"
server_key: "844e8589b1654fd2a6df2943a296ef9d"
server_key_iv: "1ad1fe8caeca636d034fcbbc994e5367"
server_key_version: "encryptionkey@0"
I20260620 07:16:57.582669 15302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:16:57.585297 15317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:57.586712 15302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:16:57.586861 15302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "aeb76548e48841cfb56807443fac2e43"
format_stamp: "Formatted at 2026-06-20 07:16:57 on dist-test-slave-j663"
server_key: "844e8589b1654fd2a6df2943a296ef9d"
server_key_iv: "1ad1fe8caeca636d034fcbbc994e5367"
server_key_version: "encryptionkey@0"
I20260620 07:16:57.586974 15302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:57.598189 15302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:57.601328 15302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:57.601539 15302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:57.608811 15302 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:34881
I20260620 07:16:57.608829 15369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:34881 every 8 connection(s)
I20260620 07:16:57.609848 15302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:57.612638 15370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:57.614655 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 15302
I20260620 07:16:57.614776 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:16:57.615052 11758 external_mini_cluster.cc:1428] Setting key ae64afa39b4f65f88cf5036988bcc5b7
I20260620 07:16:57.619231 15370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43: Bootstrap starting.
I20260620 07:16:57.621606 15370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:57.622386 15370 log.cc:824] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:16:57 dist-test-slave-j663 krb5kdc[15289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:57.624619 15370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43: No bootstrap required, opened a new log
I20260620 07:16:57.628731 15370 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb76548e48841cfb56807443fac2e43" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34881 } }
I20260620 07:16:57.629065 15370 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:57.629151 15370 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb76548e48841cfb56807443fac2e43, State: Initialized, Role: FOLLOWER
I20260620 07:16:57.629601 15370 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb76548e48841cfb56807443fac2e43" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34881 } }
I20260620 07:16:57.629767 15370 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:57.629851 15370 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:57.629973 15370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:57.630173 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:57.616502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.11.123.190:34881)
0620 07:16:57.616940 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:57.616955 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:57.616991 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:16:57.617694 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:57.617711 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:57.618032 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:57.618379 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:57.619174 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:57.620203 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:57.621022 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:57.621558 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:57.621752 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:57.624635 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:57.624644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:57.624664 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:57.624703 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:57.626947 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:57.627469 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:57.627478 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:57.627487 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:57.627658 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:57.627959 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:57.627962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:57.627963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:57.628363 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:57.628508 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:57.628832 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 07:16:57.629173 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":130,"threads_started":1}
I20260620 07:16:57.722641 15370 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb76548e48841cfb56807443fac2e43" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34881 } }
I20260620 07:16:57.723194 15370 leader_election.cc:302] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeb76548e48841cfb56807443fac2e43; no voters:
I20260620 07:16:57.723598 15370 leader_election.cc:288] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:57.723750 15375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:57.723989 15375 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [term 1 LEADER]: Becoming Leader. State: Replica: aeb76548e48841cfb56807443fac2e43, State: Running, Role: LEADER
I20260620 07:16:57.724444 15375 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb76548e48841cfb56807443fac2e43" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34881 } }
I20260620 07:16:57.725117 15370 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:57.726378 15376 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeb76548e48841cfb56807443fac2e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb76548e48841cfb56807443fac2e43" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34881 } } }
I20260620 07:16:57.726522 15376 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:57.727257 15384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:57.727631 15377 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeb76548e48841cfb56807443fac2e43. Latest consensus state: current_term: 1 leader_uuid: "aeb76548e48841cfb56807443fac2e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb76548e48841cfb56807443fac2e43" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34881 } } }
I20260620 07:16:57.727718 15377 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:57.730391 15384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:57.735707 15384 catalog_manager.cc:1347] Generated new cluster ID: 08b920b3c32d4b4c86234fb99cc8ef29
I20260620 07:16:57.735773 15384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:57.755476 15384 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:57.756830 15384 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:57.763363 15384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeb76548e48841cfb56807443fac2e43: Generated new TSK 0
I20260620 07:16:57.764289 15384 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:16:57.834429 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34881
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:57.942917 15396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:57.943166 15396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:57.943208 15396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:57.947197 15396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:57.947252 15396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:57.947314 15396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:16:57.952461 15396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:57.953516 15396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:57.954531 15396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:57.963979 15404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:57.963979 15402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:57.964013 15401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:57.964260 15396 server_base.cc:1034] running on GCE node
I20260620 07:16:57.964756 15396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:57.965467 15396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:57.966657 15396 hybrid_clock.cc:648] HybridClock initialized: now 1781939817966624 us; error 56 us; skew 500 ppm
Jun 20 07:16:57 dist-test-slave-j663 krb5kdc[15289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939817, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:57.972901 15396 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:16:57.973789 15396 webserver.cc:460] Webserver started at http://127.11.123.129:34347/ using document root <none> and password file <none>
I20260620 07:16:57.974300 15396 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:57.974361 15396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:57.974521 15396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:57.976442 15396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "3c17a55ecf544ecf9325d7418d77dd21"
format_stamp: "Formatted at 2026-06-20 07:16:57 on dist-test-slave-j663"
server_key: "d3223867c8ad5049f5e74337b74c7cf6"
server_key_iv: "44b8c158f317f52dccbed7471e314e82"
server_key_version: "encryptionkey@0"
I20260620 07:16:57.977062 15396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "3c17a55ecf544ecf9325d7418d77dd21"
format_stamp: "Formatted at 2026-06-20 07:16:57 on dist-test-slave-j663"
server_key: "d3223867c8ad5049f5e74337b74c7cf6"
server_key_iv: "44b8c158f317f52dccbed7471e314e82"
server_key_version: "encryptionkey@0"
I20260620 07:16:57.980583 15396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:16:57.983053 15411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:57.984325 15396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:57.984444 15396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "3c17a55ecf544ecf9325d7418d77dd21"
format_stamp: "Formatted at 2026-06-20 07:16:57 on dist-test-slave-j663"
server_key: "d3223867c8ad5049f5e74337b74c7cf6"
server_key_iv: "44b8c158f317f52dccbed7471e314e82"
server_key_version: "encryptionkey@0"
I20260620 07:16:57.984549 15396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:57.998584 15396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:58.001912 15396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:58.002143 15396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:58.002743 15396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:58.003568 15396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:58.003613 15396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:58.003652 15396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:58.003667 15396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:58.012672 15396 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:37447
I20260620 07:16:58.012737 15524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:37447 every 8 connection(s)
I20260620 07:16:58.013608 15396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:16:58.020661 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 15396
I20260620 07:16:58.020819 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:16:58.021147 11758 external_mini_cluster.cc:1428] Setting key f908124de2877a63dfcd691d9d6656dc
Jun 20 07:16:58 dist-test-slave-j663 krb5kdc[15289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939817, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:58.029337 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:58.015490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:37203 (local address 127.11.123.190:34881)
0620 07:16:58.015641 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:58.015645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:58.016488 (+ 843us) server_negotiation.cc:407] Connection header received
0620 07:16:58.018064 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:58.018068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:58.018130 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:58.018250 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:58.019461 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.020057 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.021727 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.022024 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.022159 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:58.024947 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:58.024953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:58.024958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:58.024987 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:58.026981 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:58.027574 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:58.027578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:58.027580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:58.027682 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:58.028020 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:58.028027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:58.028033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:58.028327 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:58.028455 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:58.028903 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 07:16:58.029064 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:16:58.029986 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:58.015851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:34881 (local address 127.11.123.129:37203)
0620 07:16:58.016318 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:58.016362 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:58.017834 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:58.018426 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:58.018433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:58.018813 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:58.019307 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:58.019321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.020213 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.020217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:58.021412 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:58.021422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.022183 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.022188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:58.022421 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:58.023086 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:58.023103 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:58.024783 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:58.027129 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:58.027137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:58.027154 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:58.027403 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:58.027788 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:58.027790 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:58.027792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:58.027917 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:58.028436 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:58.028441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:58.028656 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 07:16:58.028810 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 07:16:58.029046 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260620 07:16:58.031196 15525 heartbeater.cc:347] Connected to a master server at 127.11.123.190:34881
I20260620 07:16:58.031428 15525 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:16:58.031951 15525 heartbeater.cc:511] Master 127.11.123.190:34881 requested a full tablet report, sending...
I20260620 07:16:58.033525 15334 ts_manager.cc:194] Registered new tserver with Master: 3c17a55ecf544ecf9325d7418d77dd21 (127.11.123.129:37447)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
I20260620 07:16:58.034916 15334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:37203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:16:58.050082 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34881
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:58.156757 15530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:58.156996 15530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:58.157037 15530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:58.160981 15530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:58.161036 15530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:58.161157 15530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:16:58.166199 15530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:58.167105 15530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:58.167995 15530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:58.177173 15536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:58.177173 15535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:58.177371 15538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:58.177242 15530 server_base.cc:1034] running on GCE node
I20260620 07:16:58.177750 15530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:58.178395 15530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:58.179572 15530 hybrid_clock.cc:648] HybridClock initialized: now 1781939818179560 us; error 28 us; skew 500 ppm
Jun 20 07:16:58 dist-test-slave-j663 krb5kdc[15289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939818, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:58.182312 15530 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:16:58.183096 15530 webserver.cc:460] Webserver started at http://127.11.123.130:37007/ using document root <none> and password file <none>
I20260620 07:16:58.183634 15530 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:58.183722 15530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:58.183925 15530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:58.185971 15530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "c241dcf496414c24978597c1d5aebfaa"
format_stamp: "Formatted at 2026-06-20 07:16:58 on dist-test-slave-j663"
server_key: "b21678a9e4a512d05e216fadab413d96"
server_key_iv: "872ab423455c256f53a7e46a1153a0b4"
server_key_version: "encryptionkey@0"
I20260620 07:16:58.186520 15530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "c241dcf496414c24978597c1d5aebfaa"
format_stamp: "Formatted at 2026-06-20 07:16:58 on dist-test-slave-j663"
server_key: "b21678a9e4a512d05e216fadab413d96"
server_key_iv: "872ab423455c256f53a7e46a1153a0b4"
server_key_version: "encryptionkey@0"
I20260620 07:16:58.190224 15530 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:16:58.192682 15545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:58.193953 15530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:16:58.194092 15530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "c241dcf496414c24978597c1d5aebfaa"
format_stamp: "Formatted at 2026-06-20 07:16:58 on dist-test-slave-j663"
server_key: "b21678a9e4a512d05e216fadab413d96"
server_key_iv: "872ab423455c256f53a7e46a1153a0b4"
server_key_version: "encryptionkey@0"
I20260620 07:16:58.194203 15530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:58.212467 15530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:58.215651 15530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:58.215844 15530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:58.216391 15530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:58.217213 15530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:58.217283 15530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:58.217351 15530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:58.217401 15530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:58.225646 15530 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:45597
I20260620 07:16:58.225680 15658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:45597 every 8 connection(s)
I20260620 07:16:58.226639 15530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:16:58.235411 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 15530
I20260620 07:16:58.235524 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
Jun 20 07:16:58 dist-test-slave-j663 krb5kdc[15289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939818, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:58.235826 11758 external_mini_cluster.cc:1428] Setting key 983c5283ce8f38fa740b4587816b17bc
I20260620 07:16:58.239886 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:58.228425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:37637 (local address 127.11.123.190:34881)
0620 07:16:58.228545 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:58.228548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:58.229305 (+ 757us) server_negotiation.cc:407] Connection header received
0620 07:16:58.230674 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:58.230678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:58.230734 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:58.230833 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:58.231904 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.232426 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.233583 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.233812 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.233919 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:58.236231 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:58.236234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:58.236236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:58.236260 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:58.237837 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:58.238320 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:58.238326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:58.238331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:58.238419 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:58.238720 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:58.238726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:58.238730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:58.238973 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:58.239062 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:58.239683 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 07:16:58.239776 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:16:58.240751 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:58.228711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:34881 (local address 127.11.123.130:37637)
0620 07:16:58.229172 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:58.229214 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:58.230469 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:58.230990 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:58.230997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:58.231318 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:58.231759 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:58.231772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.232545 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.232549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:58.233462 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:58.233471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.233950 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.233954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:58.234119 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:58.234595 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:58.234610 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:58.236067 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:58.237959 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:58.237964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:58.237974 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:58.238208 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:58.238510 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:58.238513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:58.238514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:58.238633 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:58.239087 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:58.239094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:58.239423 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 07:16:58.239639 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 07:16:58.239980 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20260620 07:16:58.241899 15659 heartbeater.cc:347] Connected to a master server at 127.11.123.190:34881
I20260620 07:16:58.242143 15659 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:58.242713 15659 heartbeater.cc:511] Master 127.11.123.190:34881 requested a full tablet report, sending...
I20260620 07:16:58.243767 15334 ts_manager.cc:194] Registered new tserver with Master: c241dcf496414c24978597c1d5aebfaa (127.11.123.130:45597)
I20260620 07:16:58.244311 15334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:37637
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:16:58.263927 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34881
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:58.371372 15664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:58.371618 15664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:58.371659 15664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:58.375722 15664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:58.375777 15664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:58.375834 15664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:16:58.381258 15664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:58.382264 15664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:58.383271 15664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:58.392851 15670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:58.392905 15669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:58.393060 15664 server_base.cc:1034] running on GCE node
W20260620 07:16:58.392851 15672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:58.393535 15664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:58.394187 15664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:58.395373 15664 hybrid_clock.cc:648] HybridClock initialized: now 1781939818395343 us; error 46 us; skew 500 ppm
Jun 20 07:16:58 dist-test-slave-j663 krb5kdc[15289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939818, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:58.398052 15664 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:16:58.398996 15664 webserver.cc:460] Webserver started at http://127.11.123.131:39055/ using document root <none> and password file <none>
I20260620 07:16:58.399537 15664 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:58.399626 15664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:58.399852 15664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:58.401913 15664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "146b0ce54b0b4ae1acb9fa52ba97e3ce"
format_stamp: "Formatted at 2026-06-20 07:16:58 on dist-test-slave-j663"
server_key: "683a84eab369693222fb18cc6f3df9f1"
server_key_iv: "c2d6d869a20a78b08ee8a303a463315f"
server_key_version: "encryptionkey@0"
I20260620 07:16:58.402460 15664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "146b0ce54b0b4ae1acb9fa52ba97e3ce"
format_stamp: "Formatted at 2026-06-20 07:16:58 on dist-test-slave-j663"
server_key: "683a84eab369693222fb18cc6f3df9f1"
server_key_iv: "c2d6d869a20a78b08ee8a303a463315f"
server_key_version: "encryptionkey@0"
I20260620 07:16:58.405984 15664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:16:58.408257 15679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:58.409658 15664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:16:58.409807 15664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "146b0ce54b0b4ae1acb9fa52ba97e3ce"
format_stamp: "Formatted at 2026-06-20 07:16:58 on dist-test-slave-j663"
server_key: "683a84eab369693222fb18cc6f3df9f1"
server_key_iv: "c2d6d869a20a78b08ee8a303a463315f"
server_key_version: "encryptionkey@0"
I20260620 07:16:58.409916 15664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:58.426281 15664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:58.429302 15664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:58.429482 15664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:58.430030 15664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:16:58.430801 15664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:16:58.430871 15664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:58.430941 15664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:16:58.430992 15664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:58.439826 15664 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:38011
I20260620 07:16:58.439851 15792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:38011 every 8 connection(s)
I20260620 07:16:58.440675 15664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:16:58.449764 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 15664
I20260620 07:16:58.449887 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:16:58.450198 11758 external_mini_cluster.cc:1428] Setting key 4210aec09943431808d132e64517d3db
Jun 20 07:16:58 dist-test-slave-j663 krb5kdc[15289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939818, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:58.455329 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:58.442454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:43929 (local address 127.11.123.190:34881)
0620 07:16:58.442581 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:58.442585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:58.443576 (+ 991us) server_negotiation.cc:407] Connection header received
0620 07:16:58.445538 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:58.445541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:58.445614 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:58.445727 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:58.447026 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.447545 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.448765 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.448976 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.449067 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:58.452039 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:58.452043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:58.452046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:58.452072 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:58.453439 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:58.453908 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:58.453914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:58.453915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:58.453991 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:58.454327 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:58.454332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:58.454334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:58.454498 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:58.454600 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:58.455044 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 07:16:58.455171 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:16:58.456061 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:58.442802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:34881 (local address 127.11.123.131:43929)
0620 07:16:58.443437 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:58.443487 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:58.445294 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:58.445927 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:58.445936 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:58.446415 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:58.446896 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:58.446910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.447698 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.447701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:58.448635 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:58.448645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.449172 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.449190 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:58.449340 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:58.449979 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:58.450004 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:58.451894 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:58.453545 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:58.453551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:58.453563 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:58.453810 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:58.454095 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:58.454101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:58.454104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:58.454236 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:58.454603 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:58.454612 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:58.454831 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 07:16:58.455006 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 07:16:58.455293 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":143,"threads_started":1}
I20260620 07:16:58.457053 15793 heartbeater.cc:347] Connected to a master server at 127.11.123.190:34881
I20260620 07:16:58.457352 15793 heartbeater.cc:464] Registering TS with master...
I20260620 07:16:58.457799 15793 heartbeater.cc:511] Master 127.11.123.190:34881 requested a full tablet report, sending...
I20260620 07:16:58.458740 15334 ts_manager.cc:194] Registered new tserver with Master: 146b0ce54b0b4ae1acb9fa52ba97e3ce (127.11.123.131:38011)
I20260620 07:16:58.459372 15334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:43929
I20260620 07:16:58.464216 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:16:58 dist-test-slave-j663 krb5kdc[15289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:16:58 dist-test-slave-j663 krb5kdc[15289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.129@KRBTEST.COM
I20260620 07:16:58.488965 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:58.479869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.11.123.129:37447)
0620 07:16:58.480282 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:58.480292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:58.480314 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:16:58.480361 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:58.480366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:58.480537 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:58.480654 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:58.481338 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.482118 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.482902 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.483277 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.483360 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:58.485610 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:58.485618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:58.485629 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:58.485650 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:58.487153 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:58.487602 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:58.487607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:58.487608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:58.487697 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:58.487934 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:58.487936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:58.487937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:58.488245 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:58.488405 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:58.488614 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 07:16:58.488784 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":176,"threads_started":1}
Jun 20 07:16:58 dist-test-slave-j663 krb5kdc[15289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:58.499841 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:58.491812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.11.123.190:34881)
0620 07:16:58.491931 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:58.491936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:58.492074 (+ 138us) server_negotiation.cc:407] Connection header received
0620 07:16:58.492202 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:58.492212 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:58.492256 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:58.492340 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:58.492908 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.493495 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.494256 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:58.494474 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:58.494584 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:58.496786 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:58.496790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:58.496792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:58.496822 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:58.498196 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:58.498627 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:58.498633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:58.498637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:58.498719 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:58.498984 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:58.498988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:58.498990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:58.499206 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:58.499343 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:58.499561 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 07:16:58.499681 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 07:16:58.500222 15334 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59504
I20260620 07:16:58.500783 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 15396
I20260620 07:16:58.506232 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 15530
I20260620 07:16:58.511353 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 15664
I20260620 07:16:58.516428 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 15302
2026-06-20T07:16:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5064 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:58 dist-test-slave-j663 krb5kdc[15805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:58 dist-test-slave-j663 krb5kdc[15805](info): set up 2 sockets
Jun 20 07:16:58 dist-test-slave-j663 krb5kdc[15805](info): commencing operation
krb5kdc: starting...
W20260620 07:16:59.593320 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:16:59 dist-test-slave-j663 krb5kdc[15805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:16:59.711761 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:39201
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:39201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:59.819860 15818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:59.820147 15818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:59.820189 15818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:59.824102 15818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:59.824152 15818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:59.824172 15818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:59.824187 15818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:59.824203 15818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:59.829291 15818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:39201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:39201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:59.830238 15818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:59.831161 15818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:16:59.839075 15818 server_base.cc:1034] running on GCE node
W20260620 07:16:59.839056 15823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:59.839119 15824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:59.839037 15826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:59.839912 15818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:59.841367 15818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:59.842548 15818 hybrid_clock.cc:648] HybridClock initialized: now 1781939819842513 us; error 51 us; skew 500 ppm
Jun 20 07:16:59 dist-test-slave-j663 krb5kdc[15805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939819, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:59.846493 15818 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:16:59.847342 15818 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:59.847450 15818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:59.847725 15818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:59.849992 15818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "7163dbdb7cee4caea49aff23ed0e6e16"
format_stamp: "Formatted at 2026-06-20 07:16:59 on dist-test-slave-j663"
server_key: "3260a9e7c2077b9b50fb233ef376683a"
server_key_iv: "280f1774daf95ff73c1b3cbd0e75c106"
server_key_version: "encryptionkey@0"
I20260620 07:16:59.850548 15818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "7163dbdb7cee4caea49aff23ed0e6e16"
format_stamp: "Formatted at 2026-06-20 07:16:59 on dist-test-slave-j663"
server_key: "3260a9e7c2077b9b50fb233ef376683a"
server_key_iv: "280f1774daf95ff73c1b3cbd0e75c106"
server_key_version: "encryptionkey@0"
I20260620 07:16:59.854033 15818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:16:59.856280 15832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:59.857506 15818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:59.857641 15818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "7163dbdb7cee4caea49aff23ed0e6e16"
format_stamp: "Formatted at 2026-06-20 07:16:59 on dist-test-slave-j663"
server_key: "3260a9e7c2077b9b50fb233ef376683a"
server_key_iv: "280f1774daf95ff73c1b3cbd0e75c106"
server_key_version: "encryptionkey@0"
I20260620 07:16:59.857769 15818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:59.880676 15818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:59.883592 15818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:59.883798 15818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:59.890862 15818 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:39201
I20260620 07:16:59.891114 15884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:39201 every 8 connection(s)
I20260620 07:16:59.891532 15818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:59.894317 15885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:59.897365 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 15818
I20260620 07:16:59.897500 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:16:59.897789 11758 external_mini_cluster.cc:1428] Setting key 184a83cde82d51b17ad10914d95c4210
I20260620 07:16:59.900293 15885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16: Bootstrap starting.
I20260620 07:16:59.902830 15885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:59.903741 15885 log.cc:824] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:59.906061 15885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16: No bootstrap required, opened a new log
Jun 20 07:16:59 dist-test-slave-j663 krb5kdc[15805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:16:59.909042 15885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7163dbdb7cee4caea49aff23ed0e6e16" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39201 } }
I20260620 07:16:59.909346 15885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:59.909441 15885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7163dbdb7cee4caea49aff23ed0e6e16, State: Initialized, Role: FOLLOWER
I20260620 07:16:59.909962 15885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7163dbdb7cee4caea49aff23ed0e6e16" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39201 } }
I20260620 07:16:59.910118 15885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:59.910189 15885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:59.910291 15885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:59.911492 15885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7163dbdb7cee4caea49aff23ed0e6e16" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39201 } }
I20260620 07:16:59.911868 15885 leader_election.cc:302] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7163dbdb7cee4caea49aff23ed0e6e16; no voters:
I20260620 07:16:59.912209 15885 leader_election.cc:288] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:59.912333 15890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:59.912632 15890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [term 1 LEADER]: Becoming Leader. State: Replica: 7163dbdb7cee4caea49aff23ed0e6e16, State: Running, Role: LEADER
I20260620 07:16:59.913018 15890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7163dbdb7cee4caea49aff23ed0e6e16" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39201 } }
I20260620 07:16:59.913559 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:59.899105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45738 (local address 127.11.123.190:39201)
0620 07:16:59.899522 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:59.899545 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:59.899578 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:16:59.900185 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:59.900201 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:59.900568 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:59.900844 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:59.901778 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:59.902783 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:59.903959 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:59.904602 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:59.904726 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:59.907425 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:59.907431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:59.907443 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:59.907469 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:59.910326 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:59.910806 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:59.910813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:59.910818 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:59.910961 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:59.911228 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:59.911232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:59.911234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:59.911776 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:59.911976 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:59.912260 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 07:16:59.912580 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":161,"threads_started":1}
I20260620 07:16:59.913581 15885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:59.914808 15892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7163dbdb7cee4caea49aff23ed0e6e16. Latest consensus state: current_term: 1 leader_uuid: "7163dbdb7cee4caea49aff23ed0e6e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7163dbdb7cee4caea49aff23ed0e6e16" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39201 } } }
I20260620 07:16:59.914810 15891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7163dbdb7cee4caea49aff23ed0e6e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7163dbdb7cee4caea49aff23ed0e6e16" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39201 } } }
I20260620 07:16:59.914952 15892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:59.914978 15891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:59.915845 15899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:59.919420 15899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:59.924779 15899 catalog_manager.cc:1347] Generated new cluster ID: b9e68fbd43e145099d8d6f6d3017af4d
I20260620 07:16:59.924841 15899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:59.937587 15899 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:59.938519 15899 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:59.943203 15899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16: Generated new TSK 0
I20260620 07:16:59.943822 15899 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:16:59.971525 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39201
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:00.084892 15911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:00.085191 15911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:00.085239 15911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:00.089582 15911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:00.089639 15911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:00.089699 15911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:17:00.095276 15911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:00.096239 15911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:00.097311 15911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:00.106307 15917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:00.106315 15919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:00.106343 15916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:00.106452 15911 server_base.cc:1034] running on GCE node
I20260620 07:17:00.106925 15911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:00.107726 15911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:00.109025 15911 hybrid_clock.cc:648] HybridClock initialized: now 1781939820109003 us; error 40 us; skew 500 ppm
Jun 20 07:17:00 dist-test-slave-j663 krb5kdc[15805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939820, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:00.111932 15911 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:17:00.112587 15911 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:00.112646 15911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:00.112807 15911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:00.114902 15911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "37eecd159b27406da375cbaee10c8e16"
format_stamp: "Formatted at 2026-06-20 07:17:00 on dist-test-slave-j663"
server_key: "22fa318b5c2d77a878da161ca50f1703"
server_key_iv: "379a5ef0a3a80715fff18fab51d8c916"
server_key_version: "encryptionkey@0"
I20260620 07:17:00.115391 15911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "37eecd159b27406da375cbaee10c8e16"
format_stamp: "Formatted at 2026-06-20 07:17:00 on dist-test-slave-j663"
server_key: "22fa318b5c2d77a878da161ca50f1703"
server_key_iv: "379a5ef0a3a80715fff18fab51d8c916"
server_key_version: "encryptionkey@0"
I20260620 07:17:00.118920 15911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:17:00.121742 15925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:00.123019 15911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:17:00.123149 15911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "37eecd159b27406da375cbaee10c8e16"
format_stamp: "Formatted at 2026-06-20 07:17:00 on dist-test-slave-j663"
server_key: "22fa318b5c2d77a878da161ca50f1703"
server_key_iv: "379a5ef0a3a80715fff18fab51d8c916"
server_key_version: "encryptionkey@0"
I20260620 07:17:00.123265 15911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:00.137652 15911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:00.141059 15911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:00.141314 15911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:00.141923 15911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:00.142776 15911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:00.142853 15911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:00.142926 15911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:00.142987 15911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:00.152179 15911 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:33059
I20260620 07:17:00.152196 16038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:33059 every 8 connection(s)
I20260620 07:17:00.152741 15911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:17:00.157745 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 15911
I20260620 07:17:00.157869 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:17:00.158160 11758 external_mini_cluster.cc:1428] Setting key 08d01ba176075d8252f03c368f253d29
Jun 20 07:17:00 dist-test-slave-j663 krb5kdc[15805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939820, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:00.167855 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.154477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:53539 (local address 127.11.123.190:39201)
0620 07:17:00.154682 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:00.154686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:00.155786 (+ 1100us) server_negotiation.cc:407] Connection header received
0620 07:17:00.156674 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:00.156678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:00.156734 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:00.156841 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:00.158292 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.158940 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.160496 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.160835 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.160930 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.163864 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:00.163869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:00.163873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:00.163905 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:00.165601 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:00.166276 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:00.166279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:00.166280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:00.166352 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:00.166754 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:00.166756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:00.166757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:00.166924 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:00.166997 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:00.167620 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 07:17:00.167723 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:17:00.169068 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.154848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:39201 (local address 127.11.123.129:53539)
0620 07:17:00.155638 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:00.155678 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:00.156466 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:00.157011 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:00.157017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:00.157384 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:00.158062 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.158084 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.159061 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.159064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.160339 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.160354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.161001 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.161008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.161273 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.161899 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:00.161914 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:00.163695 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:00.165744 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:00.165753 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:00.165772 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:00.166157 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:00.166445 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:00.166450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:00.166453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:00.166642 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:00.167034 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:00.167041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:00.167371 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 07:17:00.167603 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 07:17:00.167966 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260620 07:17:00.170328 16039 heartbeater.cc:347] Connected to a master server at 127.11.123.190:39201
I20260620 07:17:00.170591 16039 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:00.171404 16039 heartbeater.cc:511] Master 127.11.123.190:39201 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
I20260620 07:17:00.173039 15849 ts_manager.cc:194] Registered new tserver with Master: 37eecd159b27406da375cbaee10c8e16 (127.11.123.129:33059)
I20260620 07:17:00.174912 15849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:53539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:17:00.188052 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39201
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:00.293416 16044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:00.293648 16044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:00.293689 16044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:00.297608 16044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:00.297657 16044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:00.297778 16044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:17:00.302851 16044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:00.303757 16044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:00.304677 16044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:00.313542 16052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:00.313576 16050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:00.313576 16049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:00.313836 16044 server_base.cc:1034] running on GCE node
I20260620 07:17:00.314286 16044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:00.314843 16044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:00.316025 16044 hybrid_clock.cc:648] HybridClock initialized: now 1781939820315973 us; error 73 us; skew 500 ppm
Jun 20 07:17:00 dist-test-slave-j663 krb5kdc[15805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939820, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:00.318632 16044 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:17:00.319258 16044 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:00.319317 16044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:00.319469 16044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:00.321468 16044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2"
format_stamp: "Formatted at 2026-06-20 07:17:00 on dist-test-slave-j663"
server_key: "e50482358298a6f6b63358a2900bbbb4"
server_key_iv: "816b86f4399caff739743912b2fcce92"
server_key_version: "encryptionkey@0"
I20260620 07:17:00.321934 16044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2"
format_stamp: "Formatted at 2026-06-20 07:17:00 on dist-test-slave-j663"
server_key: "e50482358298a6f6b63358a2900bbbb4"
server_key_iv: "816b86f4399caff739743912b2fcce92"
server_key_version: "encryptionkey@0"
I20260620 07:17:00.325357 16044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:17:00.327560 16058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:00.328680 16044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:00.328773 16044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2"
format_stamp: "Formatted at 2026-06-20 07:17:00 on dist-test-slave-j663"
server_key: "e50482358298a6f6b63358a2900bbbb4"
server_key_iv: "816b86f4399caff739743912b2fcce92"
server_key_version: "encryptionkey@0"
I20260620 07:17:00.328847 16044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:00.364519 16044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:00.367429 16044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:00.367574 16044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:00.368075 16044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:00.368813 16044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:00.368856 16044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:00.368894 16044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:00.368914 16044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:00.377785 16044 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:38125
I20260620 07:17:00.377818 16171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:38125 every 8 connection(s)
I20260620 07:17:00.378317 16044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:17:00.383569 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 16044
I20260620 07:17:00.383709 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:17:00.383987 11758 external_mini_cluster.cc:1428] Setting key cf2ea81fa8b28cdc9c197288ba21919e
Jun 20 07:17:00 dist-test-slave-j663 krb5kdc[15805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939820, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:00.391997 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.379876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:41235 (local address 127.11.123.190:39201)
0620 07:17:00.380006 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:00.380010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:00.381137 (+ 1127us) server_negotiation.cc:407] Connection header received
0620 07:17:00.382113 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:00.382117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:00.382174 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:00.382278 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:00.383396 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.384022 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.385278 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.385525 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.385635 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.388178 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:00.388184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:00.388191 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:00.388224 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:00.389971 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:00.390435 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:00.390440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:00.390443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:00.390541 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:00.390869 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:00.390875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:00.390878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:00.391133 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:00.391243 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:00.391678 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 07:17:00.391824 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:17:00.392665 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.380248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:39201 (local address 127.11.123.130:41235)
0620 07:17:00.380982 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:00.381019 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:00.381878 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:00.382462 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:00.382468 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:00.382791 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:00.383229 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.383241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.384145 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.384149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.385136 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.385165 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.385676 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.385681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.385894 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.386448 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:00.386472 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:00.388017 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:00.390078 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:00.390083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:00.390093 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:00.390332 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:00.390643 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:00.390645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:00.390646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:00.390777 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:00.391248 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:00.391252 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:00.391451 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 07:17:00.391594 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 07:17:00.391829 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
I20260620 07:17:00.393841 16172 heartbeater.cc:347] Connected to a master server at 127.11.123.190:39201
I20260620 07:17:00.394081 16172 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:00.394495 16172 heartbeater.cc:511] Master 127.11.123.190:39201 requested a full tablet report, sending...
I20260620 07:17:00.395462 15849 ts_manager.cc:194] Registered new tserver with Master: 995f9e1d2ee24d4192e24d0dbdf3c7e2 (127.11.123.130:38125)
I20260620 07:17:00.396072 15849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:41235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:17:00.412088 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39201
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:00.522233 16177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:00.522526 16177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:00.522606 16177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:00.527268 16177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:00.527343 16177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:00.527437 16177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:17:00.533392 16177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:00.534436 16177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:00.535473 16177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:00.544747 16183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:00.544749 16185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:00.544783 16182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:00.545203 16177 server_base.cc:1034] running on GCE node
I20260620 07:17:00.545552 16177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:00.546063 16177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:00.547217 16177 hybrid_clock.cc:648] HybridClock initialized: now 1781939820547207 us; error 25 us; skew 500 ppm
Jun 20 07:17:00 dist-test-slave-j663 krb5kdc[15805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939820, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:00.549727 16177 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:17:00.550349 16177 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:00.550405 16177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:00.550565 16177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:00.552383 16177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "ab62cd60e3a94cc8b04d26d28153e6e3"
format_stamp: "Formatted at 2026-06-20 07:17:00 on dist-test-slave-j663"
server_key: "8838820bf763d85c435fa774a0075c9e"
server_key_iv: "80a5bda28fae14ba52b637f191a6df1b"
server_key_version: "encryptionkey@0"
I20260620 07:17:00.552848 16177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "ab62cd60e3a94cc8b04d26d28153e6e3"
format_stamp: "Formatted at 2026-06-20 07:17:00 on dist-test-slave-j663"
server_key: "8838820bf763d85c435fa774a0075c9e"
server_key_iv: "80a5bda28fae14ba52b637f191a6df1b"
server_key_version: "encryptionkey@0"
I20260620 07:17:00.556172 16177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:17:00.558367 16191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:00.559518 16177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:00.559604 16177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "ab62cd60e3a94cc8b04d26d28153e6e3"
format_stamp: "Formatted at 2026-06-20 07:17:00 on dist-test-slave-j663"
server_key: "8838820bf763d85c435fa774a0075c9e"
server_key_iv: "80a5bda28fae14ba52b637f191a6df1b"
server_key_version: "encryptionkey@0"
I20260620 07:17:00.559680 16177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:00.572162 16177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:00.574878 16177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:00.575009 16177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:00.575469 16177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:00.576156 16177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:00.576198 16177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:00.576241 16177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:00.576262 16177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:00.584758 16177 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:46257
I20260620 07:17:00.584764 16304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:46257 every 8 connection(s)
I20260620 07:17:00.585495 16177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:17:00.588086 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 16177
I20260620 07:17:00.588204 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:17:00.588505 11758 external_mini_cluster.cc:1428] Setting key a212a821dd49f27669758d5e8a2d76b4
Jun 20 07:17:00 dist-test-slave-j663 krb5kdc[15805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939820, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:00.598531 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.587168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:59023 (local address 127.11.123.190:39201)
0620 07:17:00.587317 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:00.587328 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:00.588490 (+ 1162us) server_negotiation.cc:407] Connection header received
0620 07:17:00.589679 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:00.589683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:00.589737 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:00.589828 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:00.591091 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.591611 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.592776 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.592980 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.593053 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.595496 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:00.595499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:00.595501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:00.595523 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:00.596733 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:00.597243 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:00.597245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:00.597247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:00.597317 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:00.597611 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:00.597613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:00.597614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:00.597766 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:00.597835 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:00.598323 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 07:17:00.598415 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:17:00.599355 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.587442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:39201 (local address 127.11.123.131:59023)
0620 07:17:00.588341 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:00.588383 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:00.589446 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:00.589991 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:00.589998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:00.590383 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:00.590966 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.590980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.591718 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.591720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.592674 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.592683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.593194 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.593198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.593350 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.593862 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:00.593878 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:00.595351 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:00.596835 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:00.596839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:00.596851 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:00.597154 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:00.597397 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:00.597399 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:00.597401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:00.597528 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:00.597858 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:00.597863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:00.598110 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 07:17:00.598267 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 07:17:00.598512 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":105,"threads_started":1}
I20260620 07:17:00.600442 16305 heartbeater.cc:347] Connected to a master server at 127.11.123.190:39201
I20260620 07:17:00.600693 16305 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:00.601263 16305 heartbeater.cc:511] Master 127.11.123.190:39201 requested a full tablet report, sending...
I20260620 07:17:00.602226 15849 ts_manager.cc:194] Registered new tserver with Master: ab62cd60e3a94cc8b04d26d28153e6e3 (127.11.123.131:46257)
I20260620 07:17:00.602828 15849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:59023
I20260620 07:17:00.613915 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:17:00.623026 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.615847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.11.123.190:39201)
0620 07:17:00.615986 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:00.615990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:00.616161 (+ 171us) server_negotiation.cc:407] Connection header received
0620 07:17:00.616238 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:00.616240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:00.616283 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:00.616360 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:00.616969 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.617590 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.618377 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.618596 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.618683 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.619951 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:00.619954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:00.619956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:00.619980 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:00.621525 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:00.621915 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:00.621917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:00.621918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:00.621985 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:00.622241 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:00.622243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:00.622244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:00.622437 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:00.622535 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:00.622742 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 07:17:00.622828 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:17:00.625706 15849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:17:00.628721 15849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:17:00.642855 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.636962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:46257 (local address 127.0.0.1:36308)
0620 07:17:00.637807 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:00.637821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:00.637914 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:00.638655 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:00.638658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:00.638675 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:00.638769 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.638774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.639903 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.639907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.641474 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.641485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.642313 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.642316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.642426 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.642461 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:17:00.642565 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 07:17:00.642696 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":114,"thread_start_us":87,"threads_started":1}
I20260620 07:17:00.643018 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.637647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:38125 (local address 127.0.0.1:40356)
0620 07:17:00.637910 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:00.637922 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:00.638008 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:00.639135 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:00.639138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:00.639158 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:00.639238 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.639244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.640519 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.640522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.641614 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.641623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.642589 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.642592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.642736 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.642759 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:17:00.642830 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 07:17:00.642896 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":43,"thread_start_us":112,"threads_started":1}
I20260620 07:17:00.643293 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.637389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.11.123.131:46257)
0620 07:17:00.637768 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:00.637779 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:00.638022 (+ 243us) server_negotiation.cc:407] Connection header received
0620 07:17:00.638104 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:00.638110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:00.638335 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:00.638492 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:00.638882 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.639771 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.641610 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.642192 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.642328 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.642458 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:00.643022 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 07:17:00.643122 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":94,"threads_started":1}
I20260620 07:17:00.643303 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.636912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:33059 (local address 127.0.0.1:39560)
0620 07:17:00.637341 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:00.637385 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:00.637514 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:00.638486 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:00.638494 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:00.638517 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:00.638651 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.638658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.640235 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.640238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.641491 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.641496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.642588 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.642591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.642751 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.642790 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 07:17:00.642878 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 07:17:00.642955 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":125,"thread_start_us":61,"threads_started":1}
I20260620 07:17:00.643615 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.637044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.11.123.129:33059)
0620 07:17:00.637954 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:00.637963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:00.637987 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:17:00.638105 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:00.638110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:00.638317 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:00.638807 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:00.638945 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.640100 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.641626 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.642440 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.643066 (+ 626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.643197 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:00.643242 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 07:17:00.643324 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":81,"threads_started":1}
I20260620 07:17:00.643916 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.638368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40356 (local address 127.11.123.130:38125)
0620 07:17:00.638705 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:00.638722 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:00.638754 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:17:00.638820 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:00.638824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:00.638989 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:00.639135 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:00.639343 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.640396 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.641737 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.642479 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.642589 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.642713 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:00.642885 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 07:17:00.643766 (+ 881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":146,"threads_started":1}
I20260620 07:17:00.646335 15973 tablet_service.cc:1467] Processing CreateTablet for tablet ae35fe2a59ac46bc9f173713f5aa129f (DEFAULT_TABLE table=test-table [id=f947134855dd408eaae592f50d51c08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:00.646353 16239 tablet_service.cc:1467] Processing CreateTablet for tablet ae35fe2a59ac46bc9f173713f5aa129f (DEFAULT_TABLE table=test-table [id=f947134855dd408eaae592f50d51c08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:00.646405 16106 tablet_service.cc:1467] Processing CreateTablet for tablet ae35fe2a59ac46bc9f173713f5aa129f (DEFAULT_TABLE table=test-table [id=f947134855dd408eaae592f50d51c08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:00.647295 15973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae35fe2a59ac46bc9f173713f5aa129f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:00.647321 16106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae35fe2a59ac46bc9f173713f5aa129f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:00.647327 16239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae35fe2a59ac46bc9f173713f5aa129f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:00.653379 16322 tablet_bootstrap.cc:492] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16: Bootstrap starting.
I20260620 07:17:00.654439 16323 tablet_bootstrap.cc:492] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3: Bootstrap starting.
I20260620 07:17:00.654810 16324 tablet_bootstrap.cc:492] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2: Bootstrap starting.
I20260620 07:17:00.655567 16322 tablet_bootstrap.cc:654] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:00.656461 16324 tablet_bootstrap.cc:654] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:00.656543 16322 log.cc:824] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:00.657521 16324 log.cc:824] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:00.658568 16322 tablet_bootstrap.cc:492] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16: No bootstrap required, opened a new log
I20260620 07:17:00.658756 16322 ts_tablet_manager.cc:1397] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:17:00.658866 16323 tablet_bootstrap.cc:654] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:00.659449 16324 tablet_bootstrap.cc:492] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2: No bootstrap required, opened a new log
I20260620 07:17:00.659641 16324 ts_tablet_manager.cc:1397] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:17:00.659957 16323 log.cc:824] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:00.661978 16323 tablet_bootstrap.cc:492] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3: No bootstrap required, opened a new log
I20260620 07:17:00.662154 16323 ts_tablet_manager.cc:1397] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 07:17:00.662778 16322 raft_consensus.cc:348] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eecd159b27406da375cbaee10c8e16" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 33059 } } peers { permanent_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 46257 } } peers { permanent_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 38125 } }
I20260620 07:17:00.663069 16322 raft_consensus.cc:374] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:00.662961 16324 raft_consensus.cc:348] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eecd159b27406da375cbaee10c8e16" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 33059 } } peers { permanent_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 46257 } } peers { permanent_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 38125 } }
I20260620 07:17:00.663166 16322 raft_consensus.cc:729] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37eecd159b27406da375cbaee10c8e16, State: Initialized, Role: FOLLOWER
I20260620 07:17:00.663226 16324 raft_consensus.cc:374] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:00.663318 16324 raft_consensus.cc:729] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995f9e1d2ee24d4192e24d0dbdf3c7e2, State: Initialized, Role: FOLLOWER
I20260620 07:17:00.663600 16322 consensus_queue.cc:260] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eecd159b27406da375cbaee10c8e16" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 33059 } } peers { permanent_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 46257 } } peers { permanent_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 38125 } }
I20260620 07:17:00.663774 16324 consensus_queue.cc:260] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eecd159b27406da375cbaee10c8e16" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 33059 } } peers { permanent_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 46257 } } peers { permanent_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 38125 } }
I20260620 07:17:00.664291 16039 heartbeater.cc:503] Master 127.11.123.190:39201 was elected leader, sending a full tablet report...
I20260620 07:17:00.664572 16322 ts_tablet_manager.cc:1428] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 07:17:00.665719 16324 ts_tablet_manager.cc:1428] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260620 07:17:00.665908 16323 raft_consensus.cc:348] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eecd159b27406da375cbaee10c8e16" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 33059 } } peers { permanent_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 46257 } } peers { permanent_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 38125 } }
I20260620 07:17:00.666091 16172 heartbeater.cc:503] Master 127.11.123.190:39201 was elected leader, sending a full tablet report...
I20260620 07:17:00.666157 16323 raft_consensus.cc:374] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:00.666208 16323 raft_consensus.cc:729] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab62cd60e3a94cc8b04d26d28153e6e3, State: Initialized, Role: FOLLOWER
I20260620 07:17:00.666630 16323 consensus_queue.cc:260] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eecd159b27406da375cbaee10c8e16" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 33059 } } peers { permanent_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 46257 } } peers { permanent_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 38125 } }
I20260620 07:17:00.667399 16323 ts_tablet_manager.cc:1428] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:17:00.667721 16305 heartbeater.cc:503] Master 127.11.123.190:39201 was elected leader, sending a full tablet report...
I20260620 07:17:00.775087 16328 raft_consensus.cc:482] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:17:00.775336 16328 raft_consensus.cc:504] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eecd159b27406da375cbaee10c8e16" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 33059 } } peers { permanent_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 46257 } } peers { permanent_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 38125 } }
I20260620 07:17:00.776573 16328 leader_election.cc:288] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab62cd60e3a94cc8b04d26d28153e6e3 (127.11.123.131:46257), 995f9e1d2ee24d4192e24d0dbdf3c7e2 (127.11.123.130:38125)
I20260620 07:17:00.780437 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.776905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:49609 (local address 127.11.123.131:46257)
0620 07:17:00.777029 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:00.777034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:00.777229 (+ 195us) server_negotiation.cc:407] Connection header received
0620 07:17:00.777410 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:00.777413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:00.777459 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:00.777516 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:00.777833 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.778336 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.779454 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.779861 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.779972 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.780037 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:00.780237 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 07:17:00.780309 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:17:00.780452 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.776957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:41427 (local address 127.11.123.130:38125)
0620 07:17:00.777164 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:00.777170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:00.777283 (+ 113us) server_negotiation.cc:407] Connection header received
0620 07:17:00.777432 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:00.777434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:00.777472 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:00.777563 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:00.777868 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.778337 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.779493 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.779874 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.780006 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.780072 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:00.780285 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 07:17:00.780356 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 07:17:00.780458 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.776772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:46257 (local address 127.11.123.129:49609)
0620 07:17:00.777121 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:00.777136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:00.777283 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:00.777560 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:00.777563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:00.777578 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:00.777722 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.777728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.778451 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.778456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.779330 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.779339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.779984 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.779987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.780101 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.780128 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:17:00.780196 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 07:17:00.780294 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":18,"spinlock_wait_cycles":28544,"thread_start_us":105,"threads_started":1}
I20260620 07:17:00.780463 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.776840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:38125 (local address 127.11.123.129:41427)
0620 07:17:00.777158 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:00.777170 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:00.777305 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:00.777591 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:00.777594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:00.777630 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:00.777733 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.777737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.778447 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.778452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.779340 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:00.779348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.780012 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.780016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:00.780141 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.780158 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 07:17:00.780260 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 07:17:00.780326 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":59,"spinlock_wait_cycles":896,"thread_start_us":100,"threads_started":1}
I20260620 07:17:00.781104 16259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae35fe2a59ac46bc9f173713f5aa129f" candidate_uuid: "37eecd159b27406da375cbaee10c8e16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3" is_pre_election: true
I20260620 07:17:00.781328 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae35fe2a59ac46bc9f173713f5aa129f" candidate_uuid: "37eecd159b27406da375cbaee10c8e16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2" is_pre_election: true
I20260620 07:17:00.781380 16259 raft_consensus.cc:2454] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37eecd159b27406da375cbaee10c8e16 in term 0.
I20260620 07:17:00.781584 16126 raft_consensus.cc:2454] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37eecd159b27406da375cbaee10c8e16 in term 0.
I20260620 07:17:00.781862 15926 leader_election.cc:302] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37eecd159b27406da375cbaee10c8e16, ab62cd60e3a94cc8b04d26d28153e6e3; no voters:
I20260620 07:17:00.782109 16328 raft_consensus.cc:2781] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:17:00.782186 16328 raft_consensus.cc:482] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:17:00.782239 16328 raft_consensus.cc:3037] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:00.783325 16328 raft_consensus.cc:504] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eecd159b27406da375cbaee10c8e16" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 33059 } } peers { permanent_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 46257 } } peers { permanent_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 38125 } }
I20260620 07:17:00.783800 16328 leader_election.cc:288] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [CANDIDATE]: Term 1 election: Requested vote from peers ab62cd60e3a94cc8b04d26d28153e6e3 (127.11.123.131:46257), 995f9e1d2ee24d4192e24d0dbdf3c7e2 (127.11.123.130:38125)
I20260620 07:17:00.783984 16259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae35fe2a59ac46bc9f173713f5aa129f" candidate_uuid: "37eecd159b27406da375cbaee10c8e16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3"
I20260620 07:17:00.784085 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae35fe2a59ac46bc9f173713f5aa129f" candidate_uuid: "37eecd159b27406da375cbaee10c8e16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2"
I20260620 07:17:00.784096 16259 raft_consensus.cc:3037] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:00.784188 16126 raft_consensus.cc:3037] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:00.785422 16259 raft_consensus.cc:2454] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37eecd159b27406da375cbaee10c8e16 in term 1.
I20260620 07:17:00.785423 16126 raft_consensus.cc:2454] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37eecd159b27406da375cbaee10c8e16 in term 1.
I20260620 07:17:00.785718 15926 leader_election.cc:302] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37eecd159b27406da375cbaee10c8e16, ab62cd60e3a94cc8b04d26d28153e6e3; no voters:
I20260620 07:17:00.785887 16328 raft_consensus.cc:2781] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:00.786091 16328 raft_consensus.cc:686] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 1 LEADER]: Becoming Leader. State: Replica: 37eecd159b27406da375cbaee10c8e16, State: Running, Role: LEADER
I20260620 07:17:00.786428 16328 consensus_queue.cc:237] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eecd159b27406da375cbaee10c8e16" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 33059 } } peers { permanent_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 46257 } } peers { permanent_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 38125 } }
I20260620 07:17:00.789542 15847 catalog_manager.cc:5495] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37eecd159b27406da375cbaee10c8e16 (127.11.123.129). New cstate: current_term: 1 leader_uuid: "37eecd159b27406da375cbaee10c8e16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eecd159b27406da375cbaee10c8e16" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 33059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 46257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 38125 } health_report { overall_health: UNKNOWN } } }
I20260620 07:17:00.829344 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.825807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.11.123.129:33059)
0620 07:17:00.825935 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:00.825938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:00.825952 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:17:00.826139 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:00.826142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:00.826199 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:00.826296 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:17:00.826527 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.827150 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.827995 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:00.828220 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:00.828299 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:00.828572 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:17:00.828979 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:17:00.829153 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:00.829188 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 07:17:00.829236 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:17:00.835176 16259 raft_consensus.cc:1261] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3 [term 1 FOLLOWER]: Refusing update from remote peer 37eecd159b27406da375cbaee10c8e16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:00.835176 16126 raft_consensus.cc:1261] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2 [term 1 FOLLOWER]: Refusing update from remote peer 37eecd159b27406da375cbaee10c8e16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:00.835837 16333 consensus_queue.cc:1035] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "995f9e1d2ee24d4192e24d0dbdf3c7e2" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 38125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:00.836061 16328 consensus_queue.cc:1035] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab62cd60e3a94cc8b04d26d28153e6e3" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 46257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:17:00.836293 16306 tablet.cc:2369] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:17:00.843465 16338 mvcc.cc:205] Tried to move back new op lower bound from 7298825506134065152 to 7298825505943834624. Current Snapshot: MvccSnapshot[applied={T|T < 7298825506134065152}]
I20260620 07:17:00.852399 15848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:00.852547 15848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:00.854905 15848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7163dbdb7cee4caea49aff23ed0e6e16: Sending DeleteTablet for 3 replicas of tablet ae35fe2a59ac46bc9f173713f5aa129f
I20260620 07:17:00.855646 15973 tablet_service.cc:1514] Processing DeleteTablet for tablet ae35fe2a59ac46bc9f173713f5aa129f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:00 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:39560
I20260620 07:17:00.855646 16239 tablet_service.cc:1514] Processing DeleteTablet for tablet ae35fe2a59ac46bc9f173713f5aa129f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:00 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:36308
I20260620 07:17:00.855852 16106 tablet_service.cc:1514] Processing DeleteTablet for tablet ae35fe2a59ac46bc9f173713f5aa129f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:00 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:40356
I20260620 07:17:00.856249 16346 tablet_replica.cc:330] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16: stopping tablet replica
I20260620 07:17:00.856590 16346 raft_consensus.cc:2229] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:17:00.856644 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 15911
I20260620 07:17:00.857033 16346 raft_consensus.cc:2258] T ae35fe2a59ac46bc9f173713f5aa129f P 37eecd159b27406da375cbaee10c8e16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:00.857300 16347 tablet_replica.cc:330] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3: stopping tablet replica
I20260620 07:17:00.857568 16347 raft_consensus.cc:2229] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:00.857854 16347 raft_consensus.cc:2258] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:00.858958 16348 tablet_replica.cc:330] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2: stopping tablet replica
I20260620 07:17:00.859233 16347 ts_tablet_manager.cc:1905] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:00.859242 16348 raft_consensus.cc:2229] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:00.859503 16348 raft_consensus.cc:2258] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:00.860948 16348 ts_tablet_manager.cc:1905] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:00.862114 16347 ts_tablet_manager.cc:1918] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:00.862221 16347 log.cc:1196] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/ae35fe2a59ac46bc9f173713f5aa129f
I20260620 07:17:00.862641 16347 ts_tablet_manager.cc:1939] T ae35fe2a59ac46bc9f173713f5aa129f P ab62cd60e3a94cc8b04d26d28153e6e3: Deleting consensus metadata
I20260620 07:17:00.863701 16348 ts_tablet_manager.cc:1918] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:00.863826 16348 log.cc:1196] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/ae35fe2a59ac46bc9f173713f5aa129f
I20260620 07:17:00.863893 15833 catalog_manager.cc:4841] TS ab62cd60e3a94cc8b04d26d28153e6e3 (127.11.123.131:46257): tablet ae35fe2a59ac46bc9f173713f5aa129f (table test-table [id=f947134855dd408eaae592f50d51c08a]) successfully deleted
I20260620 07:17:00.864152 16348 ts_tablet_manager.cc:1939] T ae35fe2a59ac46bc9f173713f5aa129f P 995f9e1d2ee24d4192e24d0dbdf3c7e2: Deleting consensus metadata
I20260620 07:17:00.865057 15835 catalog_manager.cc:4841] TS 995f9e1d2ee24d4192e24d0dbdf3c7e2 (127.11.123.130:38125): tablet ae35fe2a59ac46bc9f173713f5aa129f (table test-table [id=f947134855dd408eaae592f50d51c08a]) successfully deleted
W20260620 07:17:00.865473 15836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:33059 (error 108)
I20260620 07:17:00.865710 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 16044
I20260620 07:17:00.866781 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:00.866447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:33059 (local address 127.0.0.1:39582)
0620 07:17:00.866602 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:00.866681 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 07:17:00.867014 15836 catalog_manager.cc:4584] TS 37eecd159b27406da375cbaee10c8e16 (127.11.123.129:33059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae35fe2a59ac46bc9f173713f5aa129f: Network error: Client connection negotiation failed: client connection to 127.11.123.129:33059: connect: Connection refused (error 111)
I20260620 07:17:00.871371 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 16177
I20260620 07:17:00.876739 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 15818
2026-06-20T07:17:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2360 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T07:17:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:00Z Disabled control of system clock
I20260620 07:17:00.905893 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:36695
--webserver_interface=127.11.123.190
--webserver_port=0
--builtin_ntp_servers=127.11.123.148:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:36695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 07:17:01.016187 16353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:01.016435 16353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:01.016471 16353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:01.020294 16353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:01.020349 16353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:01.020365 16353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:01.020381 16353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:01.020396 16353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:17:01.024879 16353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:36695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:36695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:01.025859 16353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:01.027150 16353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:01.035050 16358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:01.035069 16359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:01.035190 16353 server_base.cc:1034] running on GCE node
W20260620 07:17:01.035069 16361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:01.035799 16353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:01.036852 16353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:01.038069 16353 hybrid_clock.cc:648] HybridClock initialized: now 1781939821038034 us; error 39 us; skew 500 ppm
I20260620 07:17:01.039924 16353 webserver.cc:460] Webserver started at http://127.11.123.190:45439/ using document root <none> and password file <none>
I20260620 07:17:01.040567 16353 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:01.040663 16353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:01.040894 16353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:01.043001 16353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "bbd97de96da04b08824cddf93789f6c9"
format_stamp: "Formatted at 2026-06-20 07:17:01 on dist-test-slave-j663"
server_key: "727d4592286eedee94bc0cdcebb73b98"
server_key_iv: "85fced163c8628debf07b4c108b2caca"
server_key_version: "encryptionkey@0"
I20260620 07:17:01.043563 16353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "bbd97de96da04b08824cddf93789f6c9"
format_stamp: "Formatted at 2026-06-20 07:17:01 on dist-test-slave-j663"
server_key: "727d4592286eedee94bc0cdcebb73b98"
server_key_iv: "85fced163c8628debf07b4c108b2caca"
server_key_version: "encryptionkey@0"
I20260620 07:17:01.047010 16353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:17:01.049301 16367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:01.050472 16353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:01.050611 16353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "bbd97de96da04b08824cddf93789f6c9"
format_stamp: "Formatted at 2026-06-20 07:17:01 on dist-test-slave-j663"
server_key: "727d4592286eedee94bc0cdcebb73b98"
server_key_iv: "85fced163c8628debf07b4c108b2caca"
server_key_version: "encryptionkey@0"
I20260620 07:17:01.050717 16353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:01.055115 16353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:01.055681 16353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:01.055860 16353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:01.062542 16353 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:36695
I20260620 07:17:01.062547 16419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:36695 every 8 connection(s)
I20260620 07:17:01.063377 16353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:01.066300 16420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:01.067795 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 16353
I20260620 07:17:01.067936 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:17:01.068181 11758 external_mini_cluster.cc:1428] Setting key 58576fb80244c7c4be9626f6c19d11b2
I20260620 07:17:01.072719 16420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9: Bootstrap starting.
I20260620 07:17:01.073850 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.069550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.11.123.190:36695)
0620 07:17:01.070161 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:01.070180 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:01.070221 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:17:01.070958 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:01.070979 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:01.071394 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:01.071746 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:01.072421 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:01.072426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:01.072439 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:01.072452 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:01.072516 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:01.072536 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:01.072664 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:01.072738 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 07:17:01.072956 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":94,"threads_started":1}
I20260620 07:17:01.075086 16420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:01.075881 16420 log.cc:824] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:01.077646 16420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9: No bootstrap required, opened a new log
I20260620 07:17:01.080754 16420 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd97de96da04b08824cddf93789f6c9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36695 } }
I20260620 07:17:01.080971 16420 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:01.081046 16420 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd97de96da04b08824cddf93789f6c9, State: Initialized, Role: FOLLOWER
I20260620 07:17:01.081475 16420 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd97de96da04b08824cddf93789f6c9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36695 } }
I20260620 07:17:01.081635 16420 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:01.081699 16420 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:01.081787 16420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:01.082544 16420 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd97de96da04b08824cddf93789f6c9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36695 } }
I20260620 07:17:01.082851 16420 leader_election.cc:302] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbd97de96da04b08824cddf93789f6c9; no voters:
I20260620 07:17:01.083150 16420 leader_election.cc:288] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:17:01.083319 16425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:01.083546 16425 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [term 1 LEADER]: Becoming Leader. State: Replica: bbd97de96da04b08824cddf93789f6c9, State: Running, Role: LEADER
I20260620 07:17:01.083818 16425 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd97de96da04b08824cddf93789f6c9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36695 } }
I20260620 07:17:01.084306 16420 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:01.085381 16426 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbd97de96da04b08824cddf93789f6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd97de96da04b08824cddf93789f6c9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36695 } } }
I20260620 07:17:01.085506 16426 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:01.085721 16427 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbd97de96da04b08824cddf93789f6c9. Latest consensus state: current_term: 1 leader_uuid: "bbd97de96da04b08824cddf93789f6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd97de96da04b08824cddf93789f6c9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36695 } } }
I20260620 07:17:01.085866 16427 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:01.085876 16434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:01.088785 16434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:01.093753 16434 catalog_manager.cc:1347] Generated new cluster ID: c90a95ff84784dc4a8a38b6b66198b26
I20260620 07:17:01.093823 16434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:01.116981 16434 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:17:01.117303 16434 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:01.123512 16434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9: Generated new TSK 0
I20260620 07:17:01.124132 16434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:17:01.137652 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36695
--builtin_ntp_servers=127.11.123.148:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 07:17:01.245106 16444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:01.245352 16444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:01.245390 16444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:01.249356 16444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:01.249398 16444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:01.249507 16444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:17:01.254299 16444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:01.255486 16444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:01.256876 16444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:01.266096 16449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:01.266194 16452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:01.266126 16450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:01.266305 16444 server_base.cc:1034] running on GCE node
I20260620 07:17:01.266904 16444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:01.267511 16444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:01.268702 16444 hybrid_clock.cc:648] HybridClock initialized: now 1781939821268687 us; error 24 us; skew 500 ppm
I20260620 07:17:01.270463 16444 webserver.cc:460] Webserver started at http://127.11.123.129:46493/ using document root <none> and password file <none>
I20260620 07:17:01.271014 16444 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:01.271073 16444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:01.271363 16444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:01.273412 16444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "099ccdba42cb4aacb161781912631608"
format_stamp: "Formatted at 2026-06-20 07:17:01 on dist-test-slave-j663"
server_key: "16de4ce8e3962e2f016e4b1b18c8df40"
server_key_iv: "ba946285b0defef4ccb953c687a2fde1"
server_key_version: "encryptionkey@0"
I20260620 07:17:01.273949 16444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "099ccdba42cb4aacb161781912631608"
format_stamp: "Formatted at 2026-06-20 07:17:01 on dist-test-slave-j663"
server_key: "16de4ce8e3962e2f016e4b1b18c8df40"
server_key_iv: "ba946285b0defef4ccb953c687a2fde1"
server_key_version: "encryptionkey@0"
I20260620 07:17:01.277493 16444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:17:01.279894 16458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:01.281296 16444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:01.281409 16444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "099ccdba42cb4aacb161781912631608"
format_stamp: "Formatted at 2026-06-20 07:17:01 on dist-test-slave-j663"
server_key: "16de4ce8e3962e2f016e4b1b18c8df40"
server_key_iv: "ba946285b0defef4ccb953c687a2fde1"
server_key_version: "encryptionkey@0"
I20260620 07:17:01.281509 16444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:01.286280 16444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:01.286872 16444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:01.287057 16444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:01.287599 16444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:01.288511 16444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:01.288555 16444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:01.288621 16444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:01.288652 16444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:01.296846 16444 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:44637
I20260620 07:17:01.296916 16571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:44637 every 8 connection(s)
I20260620 07:17:01.297775 16444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:17:01.303433 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 16444
I20260620 07:17:01.303567 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:17:01.303772 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.299578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:36971 (local address 127.11.123.190:36695)
0620 07:17:01.299716 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:01.299720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:01.300707 (+ 987us) server_negotiation.cc:407] Connection header received
0620 07:17:01.302056 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:01.302060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:01.302105 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:01.302211 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:01.303268 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:01.303271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:01.303273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:01.303285 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:01.303306 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:01.303314 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:01.303396 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:01.303587 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 07:17:01.303633 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:17:01.303941 11758 external_mini_cluster.cc:1428] Setting key 3cf466c2c9bc04052b44613132e2f56a
I20260620 07:17:01.305044 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.299919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36695 (local address 127.11.123.129:36971)
0620 07:17:01.300555 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:01.300594 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:01.301846 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:01.302372 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:01.302379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:01.302466 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:01.302965 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:01.302967 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:01.303004 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:01.303069 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:01.303408 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:01.303418 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:01.303425 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:17:01.303614 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 07:17:01.303859 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
I20260620 07:17:01.306322 16572 heartbeater.cc:347] Connected to a master server at 127.11.123.190:36695
I20260620 07:17:01.306362 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36695
--builtin_ntp_servers=127.11.123.148:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 07:17:01.306586 16572 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:01.306990 16572 heartbeater.cc:511] Master 127.11.123.190:36695 requested a full tablet report, sending...
I20260620 07:17:01.308673 16384 ts_manager.cc:194] Registered new tserver with Master: 099ccdba42cb4aacb161781912631608 (127.11.123.129:44637)
W20260620 07:17:01.424533 16575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:01.424774 16575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:01.424813 16575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:01.428771 16575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:01.428817 16575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:01.428920 16575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:17:01.433614 16575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:01.434643 16575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:01.435966 16575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:01.445344 16581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:01.445437 16575 server_base.cc:1034] running on GCE node
W20260620 07:17:01.445333 16583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:01.445333 16580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:01.445937 16575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:01.446580 16575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:01.447762 16575 hybrid_clock.cc:648] HybridClock initialized: now 1781939821447731 us; error 44 us; skew 500 ppm
I20260620 07:17:01.449520 16575 webserver.cc:460] Webserver started at http://127.11.123.130:40215/ using document root <none> and password file <none>
I20260620 07:17:01.450132 16575 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:01.450330 16575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:01.450579 16575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:01.452489 16575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "e4af0a54096840f9adc941903e5e292e"
format_stamp: "Formatted at 2026-06-20 07:17:01 on dist-test-slave-j663"
server_key: "9f385cc0a861e32c40850a21557adfbd"
server_key_iv: "1798eda14b5bde72286de5bb0f8046b9"
server_key_version: "encryptionkey@0"
I20260620 07:17:01.453032 16575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "e4af0a54096840f9adc941903e5e292e"
format_stamp: "Formatted at 2026-06-20 07:17:01 on dist-test-slave-j663"
server_key: "9f385cc0a861e32c40850a21557adfbd"
server_key_iv: "1798eda14b5bde72286de5bb0f8046b9"
server_key_version: "encryptionkey@0"
I20260620 07:17:01.457643 16575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:17:01.460032 16589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:01.461297 16575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:01.461419 16575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "e4af0a54096840f9adc941903e5e292e"
format_stamp: "Formatted at 2026-06-20 07:17:01 on dist-test-slave-j663"
server_key: "9f385cc0a861e32c40850a21557adfbd"
server_key_iv: "1798eda14b5bde72286de5bb0f8046b9"
server_key_version: "encryptionkey@0"
I20260620 07:17:01.461529 16575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:01.465878 16575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:01.466437 16575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:01.466610 16575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:01.467116 16575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:01.467972 16575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:01.468039 16575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:01.468101 16575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:01.468148 16575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:01.477298 16575 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:39433
I20260620 07:17:01.477327 16702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:39433 every 8 connection(s)
I20260620 07:17:01.478184 16575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:17:01.484056 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 16575
I20260620 07:17:01.484180 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:17:01.484392 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.479983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:34997 (local address 127.11.123.190:36695)
0620 07:17:01.480182 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:01.480187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:01.481273 (+ 1086us) server_negotiation.cc:407] Connection header received
0620 07:17:01.482543 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:01.482546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:01.482583 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:01.482668 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:01.483808 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:01.483811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:01.483812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:01.483822 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:01.483839 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:01.483847 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:01.483956 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:01.484143 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 07:17:01.484226 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 07:17:01.484472 11758 external_mini_cluster.cc:1428] Setting key b51276ea824bc9066aaf200b7f50f597
I20260620 07:17:01.485440 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.480386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36695 (local address 127.11.123.130:34997)
0620 07:17:01.481055 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:01.481173 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:01.482360 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:01.482821 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:01.482827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:01.482925 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:01.483531 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:01.483533 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:01.483574 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:01.483647 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:01.483943 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:01.483952 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:01.483957 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:17:01.484164 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 07:17:01.484385 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20260620 07:17:01.486614 16703 heartbeater.cc:347] Connected to a master server at 127.11.123.190:36695
I20260620 07:17:01.486740 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36695
--builtin_ntp_servers=127.11.123.148:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 07:17:01.486860 16703 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:01.487246 16703 heartbeater.cc:511] Master 127.11.123.190:36695 requested a full tablet report, sending...
I20260620 07:17:01.488291 16384 ts_manager.cc:194] Registered new tserver with Master: e4af0a54096840f9adc941903e5e292e (127.11.123.130:39433)
W20260620 07:17:01.604285 16706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:01.604542 16706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:01.604583 16706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:01.608906 16706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:01.608986 16706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:01.609046 16706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:17:01.614147 16706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:01.615273 16706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:01.616676 16706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:01.625953 16712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:01.626053 16706 server_base.cc:1034] running on GCE node
W20260620 07:17:01.625996 16714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:01.625996 16711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:01.626576 16706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:01.627243 16706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:01.628425 16706 hybrid_clock.cc:648] HybridClock initialized: now 1781939821628410 us; error 29 us; skew 500 ppm
I20260620 07:17:01.630261 16706 webserver.cc:460] Webserver started at http://127.11.123.131:44233/ using document root <none> and password file <none>
I20260620 07:17:01.630837 16706 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:01.630925 16706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:01.631158 16706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:01.633392 16706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "ceb1967314834d9397bbcd07da86ef80"
format_stamp: "Formatted at 2026-06-20 07:17:01 on dist-test-slave-j663"
server_key: "6cbe061d8f9fe607b38bc464aba6a0f5"
server_key_iv: "36d9c1c515f325119187626c85e89c9b"
server_key_version: "encryptionkey@0"
I20260620 07:17:01.633955 16706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "ceb1967314834d9397bbcd07da86ef80"
format_stamp: "Formatted at 2026-06-20 07:17:01 on dist-test-slave-j663"
server_key: "6cbe061d8f9fe607b38bc464aba6a0f5"
server_key_iv: "36d9c1c515f325119187626c85e89c9b"
server_key_version: "encryptionkey@0"
I20260620 07:17:01.637571 16706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:17:01.639997 16720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:01.641332 16706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:01.641458 16706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "ceb1967314834d9397bbcd07da86ef80"
format_stamp: "Formatted at 2026-06-20 07:17:01 on dist-test-slave-j663"
server_key: "6cbe061d8f9fe607b38bc464aba6a0f5"
server_key_iv: "36d9c1c515f325119187626c85e89c9b"
server_key_version: "encryptionkey@0"
I20260620 07:17:01.641573 16706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:01.646054 16706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:01.646647 16706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:01.646831 16706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:01.647359 16706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:01.648252 16706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:01.648319 16706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:01.648388 16706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:01.648432 16706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:01.656960 16706 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:43139
I20260620 07:17:01.657025 16833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:43139 every 8 connection(s)
I20260620 07:17:01.657907 16706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:17:01.662739 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 16706
I20260620 07:17:01.662868 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:17:01.663127 11758 external_mini_cluster.cc:1428] Setting key 46942c37a5b5cc2d99a1ee4e818c8adf
I20260620 07:17:01.663826 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.659636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:35141 (local address 127.11.123.190:36695)
0620 07:17:01.659776 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:01.659779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:01.660815 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 07:17:01.662048 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:01.662051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:01.662087 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:01.662161 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:01.663338 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:01.663342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:01.663345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:01.663357 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:01.663378 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:01.663387 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:01.663502 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:01.663643 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 07:17:01.663683 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:17:01.664960 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.660055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36695 (local address 127.11.123.131:35141)
0620 07:17:01.660668 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:01.660705 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:01.661872 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:01.662311 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:01.662317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:01.662406 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:01.663053 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:01.663054 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:01.663094 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:01.663157 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:01.663504 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:01.663512 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:01.663517 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:17:01.663672 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 07:17:01.663945 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":109,"threads_started":1}
I20260620 07:17:01.666031 16834 heartbeater.cc:347] Connected to a master server at 127.11.123.190:36695
I20260620 07:17:01.666255 16834 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:01.666646 16834 heartbeater.cc:511] Master 127.11.123.190:36695 requested a full tablet report, sending...
I20260620 07:17:01.667595 16384 ts_manager.cc:194] Registered new tserver with Master: ceb1967314834d9397bbcd07da86ef80 (127.11.123.131:43139)
I20260620 07:17:01.676956 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:17:01.680466 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.678844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.11.123.190:36695)
0620 07:17:01.679006 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:01.679010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:01.679162 (+ 152us) server_negotiation.cc:407] Connection header received
0620 07:17:01.679321 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:01.679326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:01.679363 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:01.679444 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:01.680088 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:01.680092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:01.680094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:01.680105 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:01.680124 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:01.680140 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:01.680238 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:01.680304 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 07:17:01.680345 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:17:01.682984 16384 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:17:01.686282 16384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:17:01.697223 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.694014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:39433 (local address 127.0.0.1:37444)
0620 07:17:01.694343 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:01.694392 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:01.694727 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:01.695681 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:01.695688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:01.695768 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:01.696382 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:01.696384 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:01.696406 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:01.696488 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:01.696854 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:01.696862 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:01.696868 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:17:01.696993 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 07:17:01.697059 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":70,"threads_started":1}
I20260620 07:17:01.697242 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.694138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37444 (local address 127.11.123.130:39433)
0620 07:17:01.695231 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:01.695246 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:01.695273 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:17:01.695332 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:01.695338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:01.695529 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:01.695679 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:01.696635 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:01.696643 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:01.696657 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:01.696671 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:01.696726 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:01.696757 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:01.696868 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:01.697020 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 07:17:01.697113 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":122,"threads_started":1}
I20260620 07:17:01.697410 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.694100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:44637 (local address 127.0.0.1:60430)
0620 07:17:01.694677 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:01.694692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:01.694780 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:01.695805 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:01.695808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:01.695818 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:01.696707 (+ 889us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:01.696708 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:01.696717 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:01.696737 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:01.697155 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:01.697158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:01.697161 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 07:17:01.697228 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 07:17:01.697264 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":96,"threads_started":1}
I20260620 07:17:01.697530 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.694782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:43139 (local address 127.0.0.1:44294)
0620 07:17:01.695031 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:01.695044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:01.695113 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:01.696324 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:01.696327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:01.696341 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:01.696945 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:01.696947 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:01.696956 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:01.696980 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:01.697338 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:01.697341 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:01.697344 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 07:17:01.697392 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 07:17:01.697420 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":128,"threads_started":1}
I20260620 07:17:01.698009 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.694764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.11.123.129:44637)
0620 07:17:01.695357 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:01.695370 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:01.695394 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:17:01.695446 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:01.695451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:01.695657 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:01.696343 (+ 686us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:01.696936 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:01.696941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:01.696952 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:01.696962 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:01.696997 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:01.697023 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:01.697745 (+ 722us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:01.697792 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 07:17:01.697859 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":118,"threads_started":1}
I20260620 07:17:01.698431 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.695051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.11.123.131:43139)
0620 07:17:01.695399 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:01.695408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:01.695426 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:17:01.695466 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:01.695470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:01.695596 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:01.695703 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:01.697147 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:01.697153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:01.697163 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:01.697174 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:01.697213 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:01.697243 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:01.697791 (+ 548us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:01.697829 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 07:17:01.698299 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":85,"threads_started":1}
I20260620 07:17:01.700459 16768 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc9f1e3b0de4644a2835a247eedb1fd (DEFAULT_TABLE table=test-table [id=feee5905ed434a1f9c1cff750028b504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:01.700500 16506 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc9f1e3b0de4644a2835a247eedb1fd (DEFAULT_TABLE table=test-table [id=feee5905ed434a1f9c1cff750028b504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:01.700497 16637 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc9f1e3b0de4644a2835a247eedb1fd (DEFAULT_TABLE table=test-table [id=feee5905ed434a1f9c1cff750028b504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:01.701493 16768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc9f1e3b0de4644a2835a247eedb1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:01.701496 16637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc9f1e3b0de4644a2835a247eedb1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:01.701496 16506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc9f1e3b0de4644a2835a247eedb1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:01.707202 16851 tablet_bootstrap.cc:492] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80: Bootstrap starting.
I20260620 07:17:01.708788 16853 tablet_bootstrap.cc:492] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e: Bootstrap starting.
I20260620 07:17:01.709393 16852 tablet_bootstrap.cc:492] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608: Bootstrap starting.
I20260620 07:17:01.709589 16851 tablet_bootstrap.cc:654] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:01.710430 16851 log.cc:824] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:01.710897 16853 tablet_bootstrap.cc:654] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:01.711387 16852 tablet_bootstrap.cc:654] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:01.711725 16853 log.cc:824] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:01.712177 16852 log.cc:824] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:01.712294 16851 tablet_bootstrap.cc:492] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80: No bootstrap required, opened a new log
I20260620 07:17:01.712488 16851 ts_tablet_manager.cc:1397] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:17:01.713598 16853 tablet_bootstrap.cc:492] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e: No bootstrap required, opened a new log
I20260620 07:17:01.713780 16853 ts_tablet_manager.cc:1397] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:17:01.714104 16852 tablet_bootstrap.cc:492] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608: No bootstrap required, opened a new log
I20260620 07:17:01.714306 16852 ts_tablet_manager.cc:1397] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:17:01.716432 16851 raft_consensus.cc:348] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4af0a54096840f9adc941903e5e292e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39433 } } peers { permanent_uuid: "099ccdba42cb4aacb161781912631608" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44637 } } peers { permanent_uuid: "ceb1967314834d9397bbcd07da86ef80" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43139 } }
I20260620 07:17:01.716665 16851 raft_consensus.cc:374] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:01.716722 16851 raft_consensus.cc:729] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb1967314834d9397bbcd07da86ef80, State: Initialized, Role: FOLLOWER
I20260620 07:17:01.717208 16851 consensus_queue.cc:260] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4af0a54096840f9adc941903e5e292e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39433 } } peers { permanent_uuid: "099ccdba42cb4aacb161781912631608" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44637 } } peers { permanent_uuid: "ceb1967314834d9397bbcd07da86ef80" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43139 } }
I20260620 07:17:01.717774 16834 heartbeater.cc:503] Master 127.11.123.190:36695 was elected leader, sending a full tablet report...
I20260620 07:17:01.718133 16851 ts_tablet_manager.cc:1428] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 07:17:01.718261 16853 raft_consensus.cc:348] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4af0a54096840f9adc941903e5e292e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39433 } } peers { permanent_uuid: "099ccdba42cb4aacb161781912631608" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44637 } } peers { permanent_uuid: "ceb1967314834d9397bbcd07da86ef80" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43139 } }
I20260620 07:17:01.718503 16853 raft_consensus.cc:374] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:01.718556 16853 raft_consensus.cc:729] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4af0a54096840f9adc941903e5e292e, State: Initialized, Role: FOLLOWER
I20260620 07:17:01.718588 16852 raft_consensus.cc:348] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4af0a54096840f9adc941903e5e292e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39433 } } peers { permanent_uuid: "099ccdba42cb4aacb161781912631608" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44637 } } peers { permanent_uuid: "ceb1967314834d9397bbcd07da86ef80" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43139 } }
I20260620 07:17:01.718839 16852 raft_consensus.cc:374] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:01.718892 16852 raft_consensus.cc:729] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099ccdba42cb4aacb161781912631608, State: Initialized, Role: FOLLOWER
I20260620 07:17:01.718962 16853 consensus_queue.cc:260] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4af0a54096840f9adc941903e5e292e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39433 } } peers { permanent_uuid: "099ccdba42cb4aacb161781912631608" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44637 } } peers { permanent_uuid: "ceb1967314834d9397bbcd07da86ef80" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43139 } }
I20260620 07:17:01.719305 16852 consensus_queue.cc:260] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4af0a54096840f9adc941903e5e292e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39433 } } peers { permanent_uuid: "099ccdba42cb4aacb161781912631608" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44637 } } peers { permanent_uuid: "ceb1967314834d9397bbcd07da86ef80" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43139 } }
I20260620 07:17:01.719614 16703 heartbeater.cc:503] Master 127.11.123.190:36695 was elected leader, sending a full tablet report...
I20260620 07:17:01.719902 16853 ts_tablet_manager.cc:1428] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 07:17:01.720814 16572 heartbeater.cc:503] Master 127.11.123.190:36695 was elected leader, sending a full tablet report...
I20260620 07:17:01.721174 16852 ts_tablet_manager.cc:1428] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
W20260620 07:17:01.729220 16704 tablet.cc:2369] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:17:01.798938 16573 tablet.cc:2369] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:17:01.872920 16857 raft_consensus.cc:482] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:17:01.873167 16857 raft_consensus.cc:504] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4af0a54096840f9adc941903e5e292e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39433 } } peers { permanent_uuid: "099ccdba42cb4aacb161781912631608" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44637 } } peers { permanent_uuid: "ceb1967314834d9397bbcd07da86ef80" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43139 } }
I20260620 07:17:01.874325 16857 leader_election.cc:288] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4af0a54096840f9adc941903e5e292e (127.11.123.130:39433), 099ccdba42cb4aacb161781912631608 (127.11.123.129:44637)
I20260620 07:17:01.876253 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.874613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:44637 (local address 127.11.123.131:36383)
0620 07:17:01.874731 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:01.874748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:01.874834 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:01.875122 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:01.875126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:01.875141 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:01.875681 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:01.875683 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:01.875693 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:01.875720 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:01.876008 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:01.876011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:01.876014 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 07:17:01.876087 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 07:17:01.876125 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 07:17:01.876253 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.874713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:36383 (local address 127.11.123.129:44637)
0620 07:17:01.874868 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:01.874874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:01.874931 (+ 57us) server_negotiation.cc:407] Connection header received
0620 07:17:01.874972 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:01.874974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:01.875007 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:01.875099 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:01.875860 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:01.875864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:01.875866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:01.875880 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:01.875910 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:01.875918 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:01.876015 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:01.876094 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 07:17:01.876133 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:17:01.876719 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.874613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:39433 (local address 127.11.123.131:52925)
0620 07:17:01.875268 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:01.875279 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:01.875378 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:01.875624 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:01.875628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:01.875642 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:01.876115 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:01.876117 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:01.876127 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:01.876157 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:01.876466 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:01.876470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:01.876474 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:17:01.876556 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:17:01.876590 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":299,"thread_start_us":146,"threads_started":1}
I20260620 07:17:01.876705 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.874811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:52925 (local address 127.11.123.130:39433)
0620 07:17:01.874961 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:01.874967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:01.875355 (+ 388us) server_negotiation.cc:407] Connection header received
0620 07:17:01.875480 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:01.875484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:01.875524 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:01.875611 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:01.876285 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:01.876289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:01.876292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:01.876306 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:01.876330 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:01.876343 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:01.876437 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:01.876561 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 07:17:01.876601 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:17:01.876997 16526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc9f1e3b0de4644a2835a247eedb1fd" candidate_uuid: "ceb1967314834d9397bbcd07da86ef80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "099ccdba42cb4aacb161781912631608" is_pre_election: true
I20260620 07:17:01.877355 16526 raft_consensus.cc:2454] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb1967314834d9397bbcd07da86ef80 in term 0.
I20260620 07:17:01.877475 16657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc9f1e3b0de4644a2835a247eedb1fd" candidate_uuid: "ceb1967314834d9397bbcd07da86ef80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4af0a54096840f9adc941903e5e292e" is_pre_election: true
I20260620 07:17:01.877756 16657 raft_consensus.cc:2454] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb1967314834d9397bbcd07da86ef80 in term 0.
I20260620 07:17:01.877875 16722 leader_election.cc:302] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 099ccdba42cb4aacb161781912631608, ceb1967314834d9397bbcd07da86ef80; no voters:
I20260620 07:17:01.878186 16857 raft_consensus.cc:2781] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:17:01.878288 16857 raft_consensus.cc:482] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:17:01.878340 16857 raft_consensus.cc:3037] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:01.879360 16857 raft_consensus.cc:504] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4af0a54096840f9adc941903e5e292e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39433 } } peers { permanent_uuid: "099ccdba42cb4aacb161781912631608" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44637 } } peers { permanent_uuid: "ceb1967314834d9397bbcd07da86ef80" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43139 } }
I20260620 07:17:01.879715 16857 leader_election.cc:288] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [CANDIDATE]: Term 1 election: Requested vote from peers e4af0a54096840f9adc941903e5e292e (127.11.123.130:39433), 099ccdba42cb4aacb161781912631608 (127.11.123.129:44637)
I20260620 07:17:01.880095 16526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc9f1e3b0de4644a2835a247eedb1fd" candidate_uuid: "ceb1967314834d9397bbcd07da86ef80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "099ccdba42cb4aacb161781912631608"
I20260620 07:17:01.880142 16657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc9f1e3b0de4644a2835a247eedb1fd" candidate_uuid: "ceb1967314834d9397bbcd07da86ef80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4af0a54096840f9adc941903e5e292e"
I20260620 07:17:01.880249 16657 raft_consensus.cc:3037] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:01.880249 16526 raft_consensus.cc:3037] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:01.881546 16657 raft_consensus.cc:2454] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb1967314834d9397bbcd07da86ef80 in term 1.
I20260620 07:17:01.881546 16526 raft_consensus.cc:2454] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb1967314834d9397bbcd07da86ef80 in term 1.
I20260620 07:17:01.881831 16722 leader_election.cc:302] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 099ccdba42cb4aacb161781912631608, ceb1967314834d9397bbcd07da86ef80; no voters:
I20260620 07:17:01.881999 16857 raft_consensus.cc:2781] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:01.882221 16857 raft_consensus.cc:686] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 1 LEADER]: Becoming Leader. State: Replica: ceb1967314834d9397bbcd07da86ef80, State: Running, Role: LEADER
I20260620 07:17:01.882568 16857 consensus_queue.cc:237] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4af0a54096840f9adc941903e5e292e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39433 } } peers { permanent_uuid: "099ccdba42cb4aacb161781912631608" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44637 } } peers { permanent_uuid: "ceb1967314834d9397bbcd07da86ef80" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43139 } }
I20260620 07:17:01.885941 16383 catalog_manager.cc:5495] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb1967314834d9397bbcd07da86ef80 (127.11.123.131). New cstate: current_term: 1 leader_uuid: "ceb1967314834d9397bbcd07da86ef80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4af0a54096840f9adc941903e5e292e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "099ccdba42cb4aacb161781912631608" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb1967314834d9397bbcd07da86ef80" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43139 } health_report { overall_health: HEALTHY } } }
W20260620 07:17:01.908885 16835 tablet.cc:2369] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:17:01.933499 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.932052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.11.123.131:43139)
0620 07:17:01.932186 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:01.932189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:01.932202 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:17:01.932369 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:01.932371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:01.932432 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:01.932506 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:01.933162 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:01.933165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:01.933167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:01.933179 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:01.933198 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:01.933206 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:01.933283 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:01.933361 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 07:17:01.933395 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:17:01.939132 16657 raft_consensus.cc:1261] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e [term 1 FOLLOWER]: Refusing update from remote peer ceb1967314834d9397bbcd07da86ef80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:01.939132 16526 raft_consensus.cc:1261] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608 [term 1 FOLLOWER]: Refusing update from remote peer ceb1967314834d9397bbcd07da86ef80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:01.939879 16857 consensus_queue.cc:1035] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4af0a54096840f9adc941903e5e292e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:01.940129 16861 consensus_queue.cc:1035] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "099ccdba42cb4aacb161781912631608" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:01.947136 16864 mvcc.cc:205] Tried to move back new op lower bound from 7298825510655733760 to 7298825510433959936. Current Snapshot: MvccSnapshot[applied={T|T < 7298825510655733760}]
I20260620 07:17:01.948278 16865 mvcc.cc:205] Tried to move back new op lower bound from 7298825510655733760 to 7298825510433959936. Current Snapshot: MvccSnapshot[applied={T|T < 7298825510655733760}]
I20260620 07:17:01.949887 16866 mvcc.cc:205] Tried to move back new op lower bound from 7298825510655733760 to 7298825510433959936. Current Snapshot: MvccSnapshot[applied={T|T < 7298825510655733760}]
I20260620 07:17:01.956085 16383 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:01.956264 16383 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:01.958556 16383 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbd97de96da04b08824cddf93789f6c9: Sending DeleteTablet for 3 replicas of tablet 4cc9f1e3b0de4644a2835a247eedb1fd
I20260620 07:17:01.959275 16637 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc9f1e3b0de4644a2835a247eedb1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:01 UTC) from {username='slave'} at 127.0.0.1:37444
I20260620 07:17:01.959290 16506 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc9f1e3b0de4644a2835a247eedb1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:01 UTC) from {username='slave'} at 127.0.0.1:60430
I20260620 07:17:01.959522 16768 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc9f1e3b0de4644a2835a247eedb1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:01 UTC) from {username='slave'} at 127.0.0.1:44294
I20260620 07:17:01.959874 16876 tablet_replica.cc:330] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608: stopping tablet replica
I20260620 07:17:01.959937 16877 tablet_replica.cc:330] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80: stopping tablet replica
I20260620 07:17:01.960179 16876 raft_consensus.cc:2229] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:01.960211 16877 raft_consensus.cc:2229] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:17:01.960476 16876 raft_consensus.cc:2258] T 4cc9f1e3b0de4644a2835a247eedb1fd P 099ccdba42cb4aacb161781912631608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:01.960531 16877 raft_consensus.cc:2258] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:01.960562 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 16444
I20260620 07:17:01.962410 16877 ts_tablet_manager.cc:1905] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:01.965289 16875 tablet_replica.cc:330] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e: stopping tablet replica
I20260620 07:17:01.965665 16875 raft_consensus.cc:2229] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:01.965965 16877 ts_tablet_manager.cc:1918] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:01.965999 16875 raft_consensus.cc:2258] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:01.966060 16877 log.cc:1196] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/4cc9f1e3b0de4644a2835a247eedb1fd
I20260620 07:17:01.966521 16877 ts_tablet_manager.cc:1939] T 4cc9f1e3b0de4644a2835a247eedb1fd P ceb1967314834d9397bbcd07da86ef80: Deleting consensus metadata
I20260620 07:17:01.967525 16875 ts_tablet_manager.cc:1905] T 4cc9f1e3b0de4644a2835a247eedb1fd P e4af0a54096840f9adc941903e5e292e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:01.968765 16371 catalog_manager.cc:4841] TS ceb1967314834d9397bbcd07da86ef80 (127.11.123.131:43139): tablet 4cc9f1e3b0de4644a2835a247eedb1fd (table test-table [id=feee5905ed434a1f9c1cff750028b504]) successfully deleted
W20260620 07:17:01.969194 16369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:44637 (error 108)
I20260620 07:17:01.969444 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 16575
I20260620 07:17:01.973894 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.973601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:44637 (local address 127.0.0.1:60438)
0620 07:17:01.973743 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:01.973814 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 07:17:01.974133 16369 catalog_manager.cc:4584] TS 099ccdba42cb4aacb161781912631608 (127.11.123.129:44637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cc9f1e3b0de4644a2835a247eedb1fd: Network error: Client connection negotiation failed: client connection to 127.11.123.129:44637: connect: Connection refused (error 111)
I20260620 07:17:01.975684 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 16706
I20260620 07:17:01.975827 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:01.975579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:39433 (local address 127.0.0.1:37460)
0620 07:17:01.975720 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:01.975765 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.130:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 07:17:01.981024 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 16353
2026-06-20T07:17:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1103 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 07:17:02.002326 11758 mini_oidc.cc:152] Starting JWKS server
I20260620 07:17:02.003880 11758 webserver.cc:460] Webserver started at https://127.0.0.1:44607/ using document root <none> and password file <none>
I20260620 07:17:02.004068 11758 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 07:17:02.004343 11758 webserver.cc:460] Webserver started at http://127.0.0.1:33881/ using document root <none> and password file <none>
2026-06-20T07:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:02Z Disabled control of system clock
I20260620 07:17:02.013200 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:46195
--webserver_interface=127.11.123.190
--webserver_port=0
--builtin_ntp_servers=127.11.123.148:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:46195
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44607/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/testchainca.cert with env {}
W20260620 07:17:02.123229 16884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:02.123474 16884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:02.123513 16884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:02.127352 16884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:02.127399 16884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:02.127415 16884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:02.127431 16884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:02.127446 16884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:17:02.127462 16884 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:17:02.127475 16884 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44607/jwks.json
I20260620 07:17:02.132503 16884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:46195
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:46195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44607/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:02.133452 16884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:02.134356 16884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:02.142383 16892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:02.142383 16889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:02.142422 16890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:02.142768 16884 server_base.cc:1034] running on GCE node
I20260620 07:17:02.143287 16884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:02.144311 16884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:02.145466 16884 hybrid_clock.cc:648] HybridClock initialized: now 1781939822145448 us; error 29 us; skew 500 ppm
I20260620 07:17:02.147424 16884 webserver.cc:460] Webserver started at http://127.11.123.190:45783/ using document root <none> and password file <none>
I20260620 07:17:02.148020 16884 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:02.148087 16884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:02.148319 16884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:02.150374 16884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "4c544db4b147426f96a05c150ba5e576"
format_stamp: "Formatted at 2026-06-20 07:17:02 on dist-test-slave-j663"
server_key: "813a064b01dfd7b66e0ba313991ca221"
server_key_iv: "8be31c66a9ab12b346d633324e15304f"
server_key_version: "encryptionkey@0"
I20260620 07:17:02.150952 16884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "4c544db4b147426f96a05c150ba5e576"
format_stamp: "Formatted at 2026-06-20 07:17:02 on dist-test-slave-j663"
server_key: "813a064b01dfd7b66e0ba313991ca221"
server_key_iv: "8be31c66a9ab12b346d633324e15304f"
server_key_version: "encryptionkey@0"
I20260620 07:17:02.154490 16884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:17:02.156827 16898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:02.158092 16884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:02.158257 16884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "4c544db4b147426f96a05c150ba5e576"
format_stamp: "Formatted at 2026-06-20 07:17:02 on dist-test-slave-j663"
server_key: "813a064b01dfd7b66e0ba313991ca221"
server_key_iv: "8be31c66a9ab12b346d633324e15304f"
server_key_version: "encryptionkey@0"
I20260620 07:17:02.158358 16884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:02.173316 16884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:02.174027 16884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:02.174237 16884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:02.180676 16884 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:46195
I20260620 07:17:02.180694 16950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:46195 every 8 connection(s)
I20260620 07:17:02.181628 16884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:02.184746 16951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:02.188331 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 16884
I20260620 07:17:02.188452 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:17:02.188730 11758 external_mini_cluster.cc:1428] Setting key ab102c612bf5fd9c44218939b336880b
I20260620 07:17:02.190820 16951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576: Bootstrap starting.
I20260620 07:17:02.193040 16951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:02.193775 16951 log.cc:824] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:02.195534 16951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576: No bootstrap required, opened a new log
I20260620 07:17:02.198030 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:02.190164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.11.123.190:46195)
0620 07:17:02.190822 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:02.190838 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:02.190872 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:17:02.191535 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:02.191553 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:02.191942 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:02.192282 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:02.192726 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.193803 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.194706 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.195169 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.195347 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:02.196197 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:02.196202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:02.196214 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:02.196584 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:02.196637 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:02.196656 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:02.196777 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:02.196879 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 07:17:02.197175 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":137,"threads_started":1}
I20260620 07:17:02.198572 16951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c544db4b147426f96a05c150ba5e576" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46195 } }
I20260620 07:17:02.198797 16951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:02.198858 16951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c544db4b147426f96a05c150ba5e576, State: Initialized, Role: FOLLOWER
I20260620 07:17:02.199324 16951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c544db4b147426f96a05c150ba5e576" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46195 } }
I20260620 07:17:02.199460 16951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:02.199513 16951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:02.199595 16951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:02.200639 16951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c544db4b147426f96a05c150ba5e576" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46195 } }
I20260620 07:17:02.201030 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:17:02.201028 16951 leader_election.cc:302] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c544db4b147426f96a05c150ba5e576; no voters:
I20260620 07:17:02.201416 16951 leader_election.cc:288] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:17:02.201522 16956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:02.201771 16956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [term 1 LEADER]: Becoming Leader. State: Replica: 4c544db4b147426f96a05c150ba5e576, State: Running, Role: LEADER
I20260620 07:17:02.202073 16956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c544db4b147426f96a05c150ba5e576" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46195 } }
I20260620 07:17:02.202677 16951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:02.203248 16956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c544db4b147426f96a05c150ba5e576. Latest consensus state: current_term: 1 leader_uuid: "4c544db4b147426f96a05c150ba5e576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c544db4b147426f96a05c150ba5e576" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46195 } } }
I20260620 07:17:02.203362 16956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:02.203817 16958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c544db4b147426f96a05c150ba5e576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c544db4b147426f96a05c150ba5e576" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46195 } } }
I20260620 07:17:02.203964 16958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576 [sys.catalog]: This master's current role is: LEADER
W20260620 07:17:02.207238 16971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:17:02.207301 16971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:17:02.207362 16961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:02.208061 16961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:02.212908 16961 catalog_manager.cc:1347] Generated new cluster ID: d81d90e1b0a84e08ad87b117e0d68dfe
I20260620 07:17:02.212981 16961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:02.247018 16961 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:17:02.247953 16961 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:02.267293 16961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c544db4b147426f96a05c150ba5e576: Generated new TSK 0
I20260620 07:17:02.268141 16961 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 07:17:02.311108 16981 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 07:17:02.311272 16981 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 07:17:02.311898 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:02.287781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.11.123.190:46195)
0620 07:17:02.287928 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:02.287933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:02.288146 (+ 213us) server_negotiation.cc:407] Connection header received
0620 07:17:02.288375 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:02.288377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:02.288429 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:02.288515 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:17:02.288809 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.289385 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.290584 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.290790 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.290866 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:02.291214 (+ 348us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:17:02.311467 (+ 20253us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 07:17:02.311746 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 07:17:02.311985 16954 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 07:17:02.312988 16980 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:02.287704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46195 (local address 127.0.0.1:46768)
0620 07:17:02.288054 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:02.288069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:02.288212 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:02.288555 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:02.288558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:02.288568 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:17:02.288705 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:02.288711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.289483 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.289486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:02.290472 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:02.290481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.290898 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.290901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:02.290993 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:02.291042 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:17:02.311705 (+ 20663us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 07:17:02.311897 (+ 192us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.123.190:46195: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":115,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.123.190:46195: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 07:17:02.313695 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 16884
2026-06-20T07:17:02Z chronyd exiting
I20260620 07:17:02.351409 11758 test_util.cc:175] -----------------------------------------------
I20260620 07:17:02.351509 11758 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939794025788-11758-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (350 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 07:17:02.352557 11758 mini_oidc.cc:152] Starting JWKS server
I20260620 07:17:02.353475 11758 webserver.cc:460] Webserver started at https://127.0.0.1:43007/ using document root <none> and password file <none>
I20260620 07:17:02.353607 11758 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 07:17:02.353863 11758 webserver.cc:460] Webserver started at http://127.0.0.1:36279/ using document root <none> and password file <none>
2026-06-20T07:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:02Z Disabled control of system clock
I20260620 07:17:02.362633 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:36719
--webserver_interface=127.11.123.190
--webserver_port=0
--builtin_ntp_servers=127.11.123.148:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:36719
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43007/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/testchainca.cert with env {}
W20260620 07:17:02.471024 16987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:02.471276 16987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:02.471314 16987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:02.475306 16987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:02.475363 16987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:02.475378 16987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:02.475395 16987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:02.475410 16987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:17:02.475426 16987 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:17:02.475440 16987 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43007/jwks.json
I20260620 07:17:02.480798 16987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:36719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:36719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43007/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:02.481787 16987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:02.482718 16987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:02.491084 16992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:02.491057 16993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:02.491158 16987 server_base.cc:1034] running on GCE node
W20260620 07:17:02.491079 16995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:02.491969 16987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:02.493000 16987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:02.494197 16987 hybrid_clock.cc:648] HybridClock initialized: now 1781939822494168 us; error 43 us; skew 500 ppm
I20260620 07:17:02.495983 16987 webserver.cc:460] Webserver started at http://127.11.123.190:40717/ using document root <none> and password file <none>
I20260620 07:17:02.496546 16987 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:02.496637 16987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:02.496860 16987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:02.498917 16987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "b7b13644c8074f17a6d5f392f8e75519"
format_stamp: "Formatted at 2026-06-20 07:17:02 on dist-test-slave-j663"
server_key: "e85d59f00e955ead50835f1a85aad5f2"
server_key_iv: "3a654e44e99d245b0f8a6b82ace17259"
server_key_version: "encryptionkey@0"
I20260620 07:17:02.499480 16987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "b7b13644c8074f17a6d5f392f8e75519"
format_stamp: "Formatted at 2026-06-20 07:17:02 on dist-test-slave-j663"
server_key: "e85d59f00e955ead50835f1a85aad5f2"
server_key_iv: "3a654e44e99d245b0f8a6b82ace17259"
server_key_version: "encryptionkey@0"
I20260620 07:17:02.503011 16987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:17:02.505264 17001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:02.506453 16987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:02.506597 16987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "b7b13644c8074f17a6d5f392f8e75519"
format_stamp: "Formatted at 2026-06-20 07:17:02 on dist-test-slave-j663"
server_key: "e85d59f00e955ead50835f1a85aad5f2"
server_key_iv: "3a654e44e99d245b0f8a6b82ace17259"
server_key_version: "encryptionkey@0"
I20260620 07:17:02.506705 16987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:02.523109 16987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:02.523748 16987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:02.523933 16987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:02.530898 16987 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:36719
I20260620 07:17:02.530905 17053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:36719 every 8 connection(s)
I20260620 07:17:02.531765 16987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:02.534768 17054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:02.537891 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 16987
I20260620 07:17:02.538025 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:17:02.538340 11758 external_mini_cluster.cc:1428] Setting key c27773da24bf74877aa97530af80ffd8
I20260620 07:17:02.540462 17054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519: Bootstrap starting.
I20260620 07:17:02.542608 17054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:02.543288 17054 log.cc:824] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:02.544953 17054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519: No bootstrap required, opened a new log
I20260620 07:17:02.547313 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:02.539666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.11.123.190:36719)
0620 07:17:02.540259 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:02.540273 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:02.540307 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:17:02.540995 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:02.541014 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:02.541458 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:02.541801 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:02.542205 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.543190 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.544081 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.544533 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.544707 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:02.545593 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:02.545599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:02.545620 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:02.545982 (+ 362us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:02.546032 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:02.546051 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:02.546165 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:02.546277 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 07:17:02.546516 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260620 07:17:02.547887 17054 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b13644c8074f17a6d5f392f8e75519" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36719 } }
I20260620 07:17:02.548105 17054 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:02.548166 17054 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b13644c8074f17a6d5f392f8e75519, State: Initialized, Role: FOLLOWER
I20260620 07:17:02.548605 17054 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b13644c8074f17a6d5f392f8e75519" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36719 } }
I20260620 07:17:02.548731 17054 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:02.548810 17054 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:02.548920 17054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:02.549887 17054 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b13644c8074f17a6d5f392f8e75519" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36719 } }
I20260620 07:17:02.549932 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:17:02.550201 17054 leader_election.cc:302] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7b13644c8074f17a6d5f392f8e75519; no voters:
I20260620 07:17:02.550485 17054 leader_election.cc:288] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:17:02.550587 17059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:02.550801 17059 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [term 1 LEADER]: Becoming Leader. State: Replica: b7b13644c8074f17a6d5f392f8e75519, State: Running, Role: LEADER
I20260620 07:17:02.551115 17059 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b13644c8074f17a6d5f392f8e75519" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36719 } }
I20260620 07:17:02.551666 17054 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:02.552690 17060 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7b13644c8074f17a6d5f392f8e75519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b13644c8074f17a6d5f392f8e75519" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36719 } } }
I20260620 07:17:02.552831 17060 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:02.553145 17061 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7b13644c8074f17a6d5f392f8e75519. Latest consensus state: current_term: 1 leader_uuid: "b7b13644c8074f17a6d5f392f8e75519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b13644c8074f17a6d5f392f8e75519" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36719 } } }
I20260620 07:17:02.553232 17068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:02.553256 17061 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:02.556160 17068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:02.561678 17068 catalog_manager.cc:1347] Generated new cluster ID: c19c016b21664307b1d6348fe4937783
I20260620 07:17:02.561743 17068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:02.588064 17068 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:17:02.589167 17068 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:02.595279 17068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7b13644c8074f17a6d5f392f8e75519: Generated new TSK 0
I20260620 07:17:02.596025 17068 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 07:17:02.629612 17084 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 07:17:02.629725 17084 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260620 07:17:02.630447 17083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:02.613816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36719 (local address 127.0.0.1:53244)
0620 07:17:02.614116 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:02.614131 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:02.614233 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:02.614527 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:02.614530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:02.614536 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:17:02.614633 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:02.614638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.615710 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.615712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:02.616405 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:02.616412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.617008 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.617011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:02.617163 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:02.617218 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:17:02.630158 (+ 12940us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 07:17:02.630298 (+ 140us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.123.190:36719: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":119,"threads_started":1}
I20260620 07:17:02.630474 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:02.613885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.11.123.190:36719)
0620 07:17:02.614027 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:02.614032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:02.614213 (+ 181us) server_negotiation.cc:407] Connection header received
0620 07:17:02.614347 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:02.614351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:02.614405 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:02.614524 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:17:02.614739 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.615585 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.616519 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.616862 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.616967 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:02.617395 (+ 428us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:17:02.629950 (+ 12555us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 07:17:02.630320 (+ 370us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53244: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 07:17:02.630546 17057 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53244: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 07:17:02.631207 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 16987
2026-06-20T07:17:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (317 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 07:17:02.670562 11758 mini_oidc.cc:152] Starting JWKS server
I20260620 07:17:02.671435 11758 webserver.cc:460] Webserver started at https://127.0.0.1:38729/ using document root <none> and password file <none>
I20260620 07:17:02.671550 11758 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 07:17:02.671825 11758 webserver.cc:460] Webserver started at http://127.0.0.1:42699/ using document root <none> and password file <none>
2026-06-20T07:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:02Z Disabled control of system clock
I20260620 07:17:02.680359 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:39173
--webserver_interface=127.11.123.190
--webserver_port=0
--builtin_ntp_servers=127.11.123.148:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:39173
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38729/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 07:17:02.788453 17090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:02.788733 17090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:02.788801 17090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:02.792726 17090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:02.792812 17090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:02.792841 17090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:02.792879 17090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:02.792915 17090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:17:02.792946 17090 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:17:02.792977 17090 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38729/jwks.json
I20260620 07:17:02.797621 17090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:39173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38729/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:02.798686 17090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:02.800024 17090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:02.808243 17095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:02.808233 17096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:02.808333 17090 server_base.cc:1034] running on GCE node
W20260620 07:17:02.808209 17098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:02.808959 17090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:02.810099 17090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:02.811264 17090 hybrid_clock.cc:648] HybridClock initialized: now 1781939822811233 us; error 41 us; skew 500 ppm
I20260620 07:17:02.813153 17090 webserver.cc:460] Webserver started at http://127.11.123.190:38291/ using document root <none> and password file <none>
I20260620 07:17:02.813719 17090 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:02.813781 17090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:02.814035 17090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:02.816028 17090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "a84deafc736b4701bf99db005f5784e5"
format_stamp: "Formatted at 2026-06-20 07:17:02 on dist-test-slave-j663"
server_key: "4dcdb37bac7aa8d2e98668156936a4f6"
server_key_iv: "0a5cbf0aa8e3cde3107ad0edda771a9f"
server_key_version: "encryptionkey@0"
I20260620 07:17:02.816581 17090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "a84deafc736b4701bf99db005f5784e5"
format_stamp: "Formatted at 2026-06-20 07:17:02 on dist-test-slave-j663"
server_key: "4dcdb37bac7aa8d2e98668156936a4f6"
server_key_iv: "0a5cbf0aa8e3cde3107ad0edda771a9f"
server_key_version: "encryptionkey@0"
I20260620 07:17:02.820240 17090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:17:02.822691 17104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:02.823910 17090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:02.824059 17090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "a84deafc736b4701bf99db005f5784e5"
format_stamp: "Formatted at 2026-06-20 07:17:02 on dist-test-slave-j663"
server_key: "4dcdb37bac7aa8d2e98668156936a4f6"
server_key_iv: "0a5cbf0aa8e3cde3107ad0edda771a9f"
server_key_version: "encryptionkey@0"
I20260620 07:17:02.824198 17090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:02.833606 17090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:02.834263 17090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:02.834453 17090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:02.841324 17090 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:39173
I20260620 07:17:02.841360 17156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:39173 every 8 connection(s)
I20260620 07:17:02.842262 17090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:02.845445 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 17090
I20260620 07:17:02.845468 17157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:02.845584 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:17:02.845906 11758 external_mini_cluster.cc:1428] Setting key 67e79951865082f8c3ac423f431c8edc
I20260620 07:17:02.852401 17157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5: Bootstrap starting.
I20260620 07:17:02.854849 17157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:02.855351 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:02.847162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.11.123.190:39173)
0620 07:17:02.847826 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:02.847842 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:02.847878 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:17:02.848701 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:02.848736 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:02.849217 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:02.849530 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:02.849871 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.850811 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.852017 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.852468 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.852642 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:02.853549 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:02.853555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:02.853566 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:02.853957 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:02.854012 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:02.854033 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:02.854142 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:02.854340 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 07:17:02.854594 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":143,"threads_started":1}
I20260620 07:17:02.855665 17157 log.cc:824] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:02.857746 17157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5: No bootstrap required, opened a new log
I20260620 07:17:02.857975 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:17:02.861055 17157 raft_consensus.cc:348] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84deafc736b4701bf99db005f5784e5" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39173 } }
I20260620 07:17:02.861356 17157 raft_consensus.cc:374] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:02.861444 17157 raft_consensus.cc:729] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84deafc736b4701bf99db005f5784e5, State: Initialized, Role: FOLLOWER
I20260620 07:17:02.861850 17157 consensus_queue.cc:260] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84deafc736b4701bf99db005f5784e5" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39173 } }
I20260620 07:17:02.861995 17157 raft_consensus.cc:388] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:02.862083 17157 raft_consensus.cc:482] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:02.862178 17157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:02.863047 17157 raft_consensus.cc:504] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84deafc736b4701bf99db005f5784e5" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39173 } }
I20260620 07:17:02.863376 17157 leader_election.cc:302] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a84deafc736b4701bf99db005f5784e5; no voters:
I20260620 07:17:02.863667 17157 leader_election.cc:288] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:17:02.863786 17162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:02.864049 17162 raft_consensus.cc:686] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [term 1 LEADER]: Becoming Leader. State: Replica: a84deafc736b4701bf99db005f5784e5, State: Running, Role: LEADER
I20260620 07:17:02.864346 17162 consensus_queue.cc:237] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84deafc736b4701bf99db005f5784e5" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39173 } }
I20260620 07:17:02.864823 17157 sys_catalog.cc:564] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:02.866061 17164 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a84deafc736b4701bf99db005f5784e5. Latest consensus state: current_term: 1 leader_uuid: "a84deafc736b4701bf99db005f5784e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84deafc736b4701bf99db005f5784e5" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39173 } } }
I20260620 07:17:02.866214 17164 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:02.866607 17171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:02.866649 17163 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a84deafc736b4701bf99db005f5784e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84deafc736b4701bf99db005f5784e5" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39173 } } }
I20260620 07:17:02.866750 17163 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:02.869207 17171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:02.875012 17171 catalog_manager.cc:1347] Generated new cluster ID: a1a397c68fba44fd9ba2bc2c1432fe5b
I20260620 07:17:02.875075 17171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:02.895917 17171 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:17:02.897220 17171 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:02.912334 17171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a84deafc736b4701bf99db005f5784e5: Generated new TSK 0
I20260620 07:17:02.913220 17171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:17:02.934461 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:02.928496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34748 (local address 127.11.123.190:39173)
0620 07:17:02.928633 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:02.928638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:02.929373 (+ 735us) server_negotiation.cc:407] Connection header received
0620 07:17:02.929525 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:02.929528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:02.929576 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:02.929664 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:17:02.929908 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.930432 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.931643 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.931864 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.931938 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:02.932258 (+ 320us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:17:02.934086 (+ 1828us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38729/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 07:17:02.934331 (+ 245us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34748: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38729/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 07:17:02.934543 17160 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34748: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38729/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 07:17:02.937418 17186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:02.928385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:39173 (local address 127.0.0.1:34748)
0620 07:17:02.929277 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:02.929292 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:02.929405 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:02.929686 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:02.929689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:02.929696 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:17:02.929806 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:02.929813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.930543 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.930546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:02.931528 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:02.931538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:02.931979 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:02.931983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:02.932074 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:02.932104 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:17:02.934278 (+ 2174us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38729/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 07:17:02.937277 (+ 2999us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.123.190:39173: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38729/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":2809,"thread_start_us":98,"threads_started":1}
I20260620 07:17:02.938010 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 17090
E20260620 07:17:02.948202 17187 webserver.cc:533] Webserver: sslize failed:
E20260620 07:17:02.948276 17187 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T07:17:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (307 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T07:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:02Z Disabled control of system clock
I20260620 07:17:02.986002 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:33449
--webserver_interface=127.11.123.190
--webserver_port=0
--builtin_ntp_servers=127.11.123.148:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:33449
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/testchainca.cert with env {}
W20260620 07:17:03.093525 17191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:03.093760 17191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:03.093796 17191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:03.097710 17191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:03.097760 17191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:03.097775 17191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:03.097791 17191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:03.097806 17191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:17:03.097824 17191 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:17:03.097837 17191 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 07:17:03.103016 17191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:33449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:03.103897 17191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:03.104784 17191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:03.112978 17197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:03.112964 17196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:03.112994 17199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:03.113220 17191 server_base.cc:1034] running on GCE node
I20260620 07:17:03.113808 17191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:03.114876 17191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:03.116093 17191 hybrid_clock.cc:648] HybridClock initialized: now 1781939823116070 us; error 33 us; skew 500 ppm
I20260620 07:17:03.122787 17191 webserver.cc:460] Webserver started at http://127.11.123.190:44281/ using document root <none> and password file <none>
I20260620 07:17:03.123369 17191 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:03.123462 17191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:03.123694 17191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:03.125758 17191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "531841634bc8476a88d73f3fa011cc5d"
format_stamp: "Formatted at 2026-06-20 07:17:03 on dist-test-slave-j663"
server_key: "628a6fee969b23f5d73854d7233a000d"
server_key_iv: "2a435db4b40c0287eaab0607007cf75f"
server_key_version: "encryptionkey@0"
I20260620 07:17:03.126325 17191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "531841634bc8476a88d73f3fa011cc5d"
format_stamp: "Formatted at 2026-06-20 07:17:03 on dist-test-slave-j663"
server_key: "628a6fee969b23f5d73854d7233a000d"
server_key_iv: "2a435db4b40c0287eaab0607007cf75f"
server_key_version: "encryptionkey@0"
I20260620 07:17:03.129943 17191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:17:03.132411 17205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:03.133627 17191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:03.133764 17191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "531841634bc8476a88d73f3fa011cc5d"
format_stamp: "Formatted at 2026-06-20 07:17:03 on dist-test-slave-j663"
server_key: "628a6fee969b23f5d73854d7233a000d"
server_key_iv: "2a435db4b40c0287eaab0607007cf75f"
server_key_version: "encryptionkey@0"
I20260620 07:17:03.133875 17191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:03.144776 17191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:03.145390 17191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:03.145577 17191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:03.152271 17191 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:33449
I20260620 07:17:03.152289 17257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:33449 every 8 connection(s)
I20260620 07:17:03.153159 17191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:03.155849 17258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:03.160952 17258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d: Bootstrap starting.
I20260620 07:17:03.161477 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 17191
I20260620 07:17:03.161595 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:17:03.161877 11758 external_mini_cluster.cc:1428] Setting key 48a045c4bcb109dffd127efd09102a27
I20260620 07:17:03.163636 17258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:03.164335 17258 log.cc:824] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:03.166143 17258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d: No bootstrap required, opened a new log
I20260620 07:17:03.169428 17258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531841634bc8476a88d73f3fa011cc5d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33449 } }
I20260620 07:17:03.169653 17258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:03.169718 17258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531841634bc8476a88d73f3fa011cc5d, State: Initialized, Role: FOLLOWER
I20260620 07:17:03.170215 17258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531841634bc8476a88d73f3fa011cc5d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33449 } }
I20260620 07:17:03.170367 17258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:03.170449 17258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:03.170567 17258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:03.171079 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:03.163072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34894 (local address 127.11.123.190:33449)
0620 07:17:03.163692 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:03.163705 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:03.163768 (+ 63us) server_negotiation.cc:407] Connection header received
0620 07:17:03.164375 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:03.164390 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:03.164888 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:03.165312 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:03.165724 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:03.166727 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:03.167597 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:03.168081 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:03.168253 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:03.169064 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:03.169070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:03.169132 (+ 62us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:03.169504 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:03.169558 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:03.169578 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:03.169687 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:03.169843 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 07:17:03.170092 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260620 07:17:03.171594 17258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531841634bc8476a88d73f3fa011cc5d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33449 } }
I20260620 07:17:03.171929 17258 leader_election.cc:302] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 531841634bc8476a88d73f3fa011cc5d; no voters:
I20260620 07:17:03.172233 17258 leader_election.cc:288] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:17:03.172319 17263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:03.172509 17263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [term 1 LEADER]: Becoming Leader. State: Replica: 531841634bc8476a88d73f3fa011cc5d, State: Running, Role: LEADER
I20260620 07:17:03.172782 17263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531841634bc8476a88d73f3fa011cc5d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33449 } }
I20260620 07:17:03.173563 17258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:03.174084 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:17:03.173985 17263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 531841634bc8476a88d73f3fa011cc5d. Latest consensus state: current_term: 1 leader_uuid: "531841634bc8476a88d73f3fa011cc5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531841634bc8476a88d73f3fa011cc5d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33449 } } }
I20260620 07:17:03.174150 17263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:03.174405 17264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "531841634bc8476a88d73f3fa011cc5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531841634bc8476a88d73f3fa011cc5d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33449 } } }
I20260620 07:17:03.174489 17264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d [sys.catalog]: This master's current role is: LEADER
W20260620 07:17:03.178282 17278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:17:03.178377 17278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:17:03.178465 17267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:03.179270 17267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:03.184372 17267 catalog_manager.cc:1347] Generated new cluster ID: 41b38199ff434633a2dbb039d271ef40
I20260620 07:17:03.184444 17267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:03.191026 17267 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:17:03.191876 17267 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:03.199574 17267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 531841634bc8476a88d73f3fa011cc5d: Generated new TSK 0
I20260620 07:17:03.200224 17267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:17:03.362396 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:03.212886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.11.123.190:33449)
0620 07:17:03.213023 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:03.213028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:03.213220 (+ 192us) server_negotiation.cc:407] Connection header received
0620 07:17:03.213370 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:03.213372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:03.213417 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:03.213497 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:17:03.213764 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:03.214290 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:03.215492 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:03.215709 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:03.215790 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:03.216112 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:17:03.361983 (+145871us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 07:17:03.362272 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34910: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 07:17:03.362504 17261 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34910: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 07:17:03.362627 17287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:03.212820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33449 (local address 127.0.0.1:34910)
0620 07:17:03.213117 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:03.213132 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:03.213241 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:03.213525 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:03.213528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:03.213537 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:17:03.213662 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:03.213669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:03.214398 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:03.214402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:03.215374 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:03.215383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:03.215829 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:03.215833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:03.215921 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:03.215954 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:17:03.362232 (+146278us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 07:17:03.362419 (+ 187us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.123.190:33449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":116,"threads_started":1}
I20260620 07:17:03.363220 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 17191
2026-06-20T07:17:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (404 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (124 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:17:03 dist-test-slave-j663 krb5kdc[17299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:17:03 dist-test-slave-j663 krb5kdc[17299](info): set up 2 sockets
Jun 20 07:17:03 dist-test-slave-j663 krb5kdc[17299](info): commencing operation
krb5kdc: starting...
W20260620 07:17:04.657498 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:17:04 dist-test-slave-j663 krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:17:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:17:04.778307 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:33869
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:33869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:04.903417 17312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:04.903684 17312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:04.903724 17312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:04.907724 17312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:04.907783 17312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:04.907804 17312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:04.907819 17312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:04.907864 17312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:17:04.913164 17312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:33869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:33869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:04.914314 17312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:04.915340 17312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:04.925163 17317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:04.925244 17320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:04.925169 17318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:04.925688 17312 server_base.cc:1034] running on GCE node
I20260620 07:17:04.926395 17312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:04.927464 17312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:04.928643 17312 hybrid_clock.cc:648] HybridClock initialized: now 1781939824928613 us; error 40 us; skew 500 ppm
Jun 20 07:17:04 dist-test-slave-j663 krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939824, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:04.931438 17312 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:17:04.932348 17312 webserver.cc:460] Webserver started at http://127.11.123.190:45089/ using document root <none> and password file <none>
I20260620 07:17:04.932927 17312 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:04.933022 17312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:04.933277 17312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:04.935366 17312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "286919d5bcbc472781475be8f37b4419"
format_stamp: "Formatted at 2026-06-20 07:17:04 on dist-test-slave-j663"
server_key: "9df2ad9e1c117609b64cb29b3c98c6f3"
server_key_iv: "814361d0d0ad87fca4282affd318b001"
server_key_version: "encryptionkey@0"
I20260620 07:17:04.935950 17312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "286919d5bcbc472781475be8f37b4419"
format_stamp: "Formatted at 2026-06-20 07:17:04 on dist-test-slave-j663"
server_key: "9df2ad9e1c117609b64cb29b3c98c6f3"
server_key_iv: "814361d0d0ad87fca4282affd318b001"
server_key_version: "encryptionkey@0"
I20260620 07:17:04.940124 17312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:17:04.942510 17327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:04.943836 17312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:04.943966 17312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "286919d5bcbc472781475be8f37b4419"
format_stamp: "Formatted at 2026-06-20 07:17:04 on dist-test-slave-j663"
server_key: "9df2ad9e1c117609b64cb29b3c98c6f3"
server_key_iv: "814361d0d0ad87fca4282affd318b001"
server_key_version: "encryptionkey@0"
I20260620 07:17:04.944075 17312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:04.955101 17312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:04.958165 17312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:04.958360 17312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:04.965307 17312 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:33869
I20260620 07:17:04.965334 17379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:33869 every 8 connection(s)
I20260620 07:17:04.966257 17312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:04.968964 17380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:04.974279 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 17312
I20260620 07:17:04.974387 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:17:04.974658 11758 external_mini_cluster.cc:1428] Setting key b7d887b4363b5c239c6698b116b2ecd9
I20260620 07:17:04.974905 17380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419: Bootstrap starting.
I20260620 07:17:04.977461 17380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:04.978201 17380 log.cc:824] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:04.980041 17380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419: No bootstrap required, opened a new log
Jun 20 07:17:04 dist-test-slave-j663 krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:04.983477 17380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286919d5bcbc472781475be8f37b4419" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33869 } }
I20260620 07:17:04.983709 17380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:04.983770 17380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286919d5bcbc472781475be8f37b4419, State: Initialized, Role: FOLLOWER
I20260620 07:17:04.984237 17380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286919d5bcbc472781475be8f37b4419" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33869 } }
I20260620 07:17:04.984385 17380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:04.984476 17380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:04.984570 17380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:04.985776 17380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286919d5bcbc472781475be8f37b4419" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33869 } }
I20260620 07:17:04.986166 17380 leader_election.cc:302] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 286919d5bcbc472781475be8f37b4419; no voters:
I20260620 07:17:04.986519 17380 leader_election.cc:288] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:17:04.986608 17385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:04.986914 17385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [term 1 LEADER]: Becoming Leader. State: Replica: 286919d5bcbc472781475be8f37b4419, State: Running, Role: LEADER
I20260620 07:17:04.987277 17385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286919d5bcbc472781475be8f37b4419" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33869 } }
I20260620 07:17:04.987762 17380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:04.988946 17386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "286919d5bcbc472781475be8f37b4419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286919d5bcbc472781475be8f37b4419" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33869 } } }
I20260620 07:17:04.989064 17386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:04.989394 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:04.976000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38688 (local address 127.11.123.190:33869)
0620 07:17:04.976458 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:04.976473 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:04.976508 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:17:04.977206 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:04.977226 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:04.977588 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:04.977944 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:04.978729 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:04.979718 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:04.980628 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:04.981186 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:04.981354 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:04.983845 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:04.983868 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:04.983888 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:04.983920 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:04.985997 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:04.986449 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:04.986457 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:04.986477 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:04.986619 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:04.986964 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:04.986968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:04.986971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:04.987488 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:04.987681 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:04.987998 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 07:17:04.988295 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":150,"threads_started":1}
I20260620 07:17:04.989362 17386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286919d5bcbc472781475be8f37b4419. Latest consensus state: current_term: 1 leader_uuid: "286919d5bcbc472781475be8f37b4419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286919d5bcbc472781475be8f37b4419" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33869 } } }
I20260620 07:17:04.989465 17386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:04.989730 17394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:04.993433 17394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:04.999140 17394 catalog_manager.cc:1347] Generated new cluster ID: 74fae2a8a016453a9d9bc3cdd8825807
I20260620 07:17:04.999248 17394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:05.012696 17394 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:17:05.013727 17394 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:05.022620 17394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 286919d5bcbc472781475be8f37b4419: Generated new TSK 0
I20260620 07:17:05.023216 17394 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:17:05.053964 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33869
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:05.164146 17406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:05.164389 17406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:05.164431 17406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:05.168610 17406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:05.168673 17406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:05.168758 17406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:17:05.174046 17406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:05.175135 17406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:05.176191 17406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:05.185745 17411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:05.185883 17406 server_base.cc:1034] running on GCE node
W20260620 07:17:05.185832 17412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:05.185745 17414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:05.186496 17406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:05.187124 17406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:05.188279 17406 hybrid_clock.cc:648] HybridClock initialized: now 1781939825188265 us; error 26 us; skew 500 ppm
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:05.191351 17406 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:17:05.192415 17406 webserver.cc:460] Webserver started at http://127.11.123.129:40445/ using document root <none> and password file <none>
I20260620 07:17:05.193020 17406 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:05.193138 17406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:05.193387 17406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:05.195416 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "21a3152e0e724c708cabddd337cf0c6b"
format_stamp: "Formatted at 2026-06-20 07:17:05 on dist-test-slave-j663"
server_key: "91a3802cbfc28913b6a7a1a9ca481e65"
server_key_iv: "416e995f1d52fe1e9e717ea3e20d3c4b"
server_key_version: "encryptionkey@0"
I20260620 07:17:05.195976 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "21a3152e0e724c708cabddd337cf0c6b"
format_stamp: "Formatted at 2026-06-20 07:17:05 on dist-test-slave-j663"
server_key: "91a3802cbfc28913b6a7a1a9ca481e65"
server_key_iv: "416e995f1d52fe1e9e717ea3e20d3c4b"
server_key_version: "encryptionkey@0"
I20260620 07:17:05.199791 17406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 07:17:05.202342 17421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:05.203619 17406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:05.203761 17406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "21a3152e0e724c708cabddd337cf0c6b"
format_stamp: "Formatted at 2026-06-20 07:17:05 on dist-test-slave-j663"
server_key: "91a3802cbfc28913b6a7a1a9ca481e65"
server_key_iv: "416e995f1d52fe1e9e717ea3e20d3c4b"
server_key_version: "encryptionkey@0"
I20260620 07:17:05.203877 17406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:05.220135 17406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:05.223445 17406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:05.223649 17406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:05.224233 17406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:05.225061 17406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:05.225198 17406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:05.225271 17406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:05.225322 17406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:05.234025 17406 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:44753
I20260620 07:17:05.234056 17534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:44753 every 8 connection(s)
I20260620 07:17:05.234882 17406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:17:05.240063 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 17406
I20260620 07:17:05.240211 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:17:05.240509 11758 external_mini_cluster.cc:1428] Setting key bb89aa0695e8a3399c8d8b83e062344f
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:05.249663 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:05.236598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:50521 (local address 127.11.123.190:33869)
0620 07:17:05.236749 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:05.236753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:05.237616 (+ 863us) server_negotiation.cc:407] Connection header received
0620 07:17:05.238923 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:05.238926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:05.238977 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:05.239059 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:05.240261 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.240982 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.242513 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.242932 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.243056 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:05.245530 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:05.245538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:05.245544 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:05.245578 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:05.247351 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:05.247869 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:05.247875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:05.247880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:05.247974 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:05.248403 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:05.248407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:05.248411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:05.248654 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:05.248767 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:05.249317 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 07:17:05.249453 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:17:05.250409 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:05.236958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33869 (local address 127.11.123.129:50521)
0620 07:17:05.237466 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:05.237506 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:05.238726 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:05.239254 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:05.239261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:05.239589 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:05.240069 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:05.240082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.241238 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.241242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:05.242385 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:05.242394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.243083 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.243086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:05.243244 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:05.243874 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:05.243889 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:05.245367 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:05.247465 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:05.247470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:05.247482 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:05.247753 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:05.248139 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:05.248142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:05.248143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:05.248299 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:05.248763 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:05.248768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:05.249004 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 07:17:05.249237 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 07:17:05.249488 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":187,"threads_started":1}
I20260620 07:17:05.251646 17535 heartbeater.cc:347] Connected to a master server at 127.11.123.190:33869
I20260620 07:17:05.251891 17535 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:05.252359 17535 heartbeater.cc:511] Master 127.11.123.190:33869 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
I20260620 07:17:05.253834 17344 ts_manager.cc:194] Registered new tserver with Master: 21a3152e0e724c708cabddd337cf0c6b (127.11.123.129:44753)
I20260620 07:17:05.254941 17344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:50521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:17:05.268064 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33869
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:05.376418 17540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:05.376681 17540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:05.376724 17540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:05.380726 17540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:05.380784 17540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:05.380875 17540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:17:05.386538 17540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:05.387645 17540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:05.388746 17540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:05.398622 17545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:05.398587 17546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:05.398718 17540 server_base.cc:1034] running on GCE node
W20260620 07:17:05.398583 17548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:05.399277 17540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:05.399941 17540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:05.401187 17540 hybrid_clock.cc:648] HybridClock initialized: now 1781939825401159 us; error 30 us; skew 500 ppm
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:05.403771 17540 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:17:05.404629 17540 webserver.cc:460] Webserver started at http://127.11.123.130:33797/ using document root <none> and password file <none>
I20260620 07:17:05.405247 17540 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:05.405339 17540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:05.405566 17540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:05.407687 17540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "a991e44de0be48ea8bb008adda813c49"
format_stamp: "Formatted at 2026-06-20 07:17:05 on dist-test-slave-j663"
server_key: "c19048c1d27977fcc86228c7923b8e77"
server_key_iv: "13e9760eddd5c4675b63cfde5e582317"
server_key_version: "encryptionkey@0"
I20260620 07:17:05.408337 17540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "a991e44de0be48ea8bb008adda813c49"
format_stamp: "Formatted at 2026-06-20 07:17:05 on dist-test-slave-j663"
server_key: "c19048c1d27977fcc86228c7923b8e77"
server_key_iv: "13e9760eddd5c4675b63cfde5e582317"
server_key_version: "encryptionkey@0"
I20260620 07:17:05.411902 17540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:17:05.414300 17555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:05.415457 17540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:05.415587 17540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "a991e44de0be48ea8bb008adda813c49"
format_stamp: "Formatted at 2026-06-20 07:17:05 on dist-test-slave-j663"
server_key: "c19048c1d27977fcc86228c7923b8e77"
server_key_iv: "13e9760eddd5c4675b63cfde5e582317"
server_key_version: "encryptionkey@0"
I20260620 07:17:05.415695 17540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:05.449918 17540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:05.453040 17540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:05.453308 17540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:05.453898 17540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:05.454748 17540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:05.454795 17540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:05.454869 17540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:05.454907 17540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:05.463812 17540 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:45537
I20260620 07:17:05.463881 17668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:45537 every 8 connection(s)
I20260620 07:17:05.464737 17540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:17:05.474354 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 17540
I20260620 07:17:05.474457 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:17:05.474753 11758 external_mini_cluster.cc:1428] Setting key ebba62ebf8535dd6e24802edb811a45d
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:05.479741 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:05.466484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:58259 (local address 127.11.123.190:33869)
0620 07:17:05.466618 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:05.466622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:05.467384 (+ 762us) server_negotiation.cc:407] Connection header received
0620 07:17:05.469202 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:05.469206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:05.469263 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:05.469355 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:05.470986 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.471870 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.472998 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.473282 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.473363 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:05.475997 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:05.476002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:05.476028 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:05.476074 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:05.477611 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:05.478126 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:05.478132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:05.478135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:05.478232 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:05.478608 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:05.478611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:05.478615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:05.478848 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:05.478929 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:05.479445 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 07:17:05.479566 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:17:05.480618 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:05.466787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33869 (local address 127.11.123.130:58259)
0620 07:17:05.467226 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:05.467266 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:05.468894 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:05.469603 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:05.469612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:05.470111 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:05.470804 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:05.470825 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.471993 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.471996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:05.472892 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:05.472901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.473418 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.473420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:05.473564 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:05.474175 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:05.474191 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:05.475843 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:05.477754 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:05.477760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:05.477775 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:05.478027 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:05.478354 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:05.478356 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:05.478358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:05.478501 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:05.478961 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:05.478966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:05.479201 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 07:17:05.479403 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 07:17:05.479719 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
I20260620 07:17:05.481802 17669 heartbeater.cc:347] Connected to a master server at 127.11.123.190:33869
I20260620 07:17:05.482048 17669 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:05.482514 17669 heartbeater.cc:511] Master 127.11.123.190:33869 requested a full tablet report, sending...
I20260620 07:17:05.483536 17344 ts_manager.cc:194] Registered new tserver with Master: a991e44de0be48ea8bb008adda813c49 (127.11.123.130:45537)
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
I20260620 07:17:05.484112 17344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:58259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:17:05.504158 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33869
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:05.609925 17674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:05.610164 17674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:05.610237 17674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:05.614218 17674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:05.614271 17674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:05.614362 17674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:17:05.619434 17674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:05.620426 17674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:05.621428 17674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:05.630487 17679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:05.630543 17680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:05.630764 17674 server_base.cc:1034] running on GCE node
W20260620 07:17:05.630817 17682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:05.631153 17674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:05.631784 17674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:05.632972 17674 hybrid_clock.cc:648] HybridClock initialized: now 1781939825632951 us; error 30 us; skew 500 ppm
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:05.635666 17674 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:17:05.636453 17674 webserver.cc:460] Webserver started at http://127.11.123.131:37587/ using document root <none> and password file <none>
I20260620 07:17:05.636982 17674 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:05.637066 17674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:05.637313 17674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:05.639588 17674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "3d1f242c8ac34ed9a84efc0388de63b9"
format_stamp: "Formatted at 2026-06-20 07:17:05 on dist-test-slave-j663"
server_key: "b5de92a21d54bf81e83c76d0f1eec894"
server_key_iv: "171038835daf4feb7fad17414ad234af"
server_key_version: "encryptionkey@0"
I20260620 07:17:05.640177 17674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "3d1f242c8ac34ed9a84efc0388de63b9"
format_stamp: "Formatted at 2026-06-20 07:17:05 on dist-test-slave-j663"
server_key: "b5de92a21d54bf81e83c76d0f1eec894"
server_key_iv: "171038835daf4feb7fad17414ad234af"
server_key_version: "encryptionkey@0"
I20260620 07:17:05.643780 17674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 07:17:05.646515 17689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:05.647850 17674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:05.647974 17674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "3d1f242c8ac34ed9a84efc0388de63b9"
format_stamp: "Formatted at 2026-06-20 07:17:05 on dist-test-slave-j663"
server_key: "b5de92a21d54bf81e83c76d0f1eec894"
server_key_iv: "171038835daf4feb7fad17414ad234af"
server_key_version: "encryptionkey@0"
I20260620 07:17:05.648231 17674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:05.684263 17674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:05.687470 17674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:05.687667 17674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:05.688230 17674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:05.689157 17674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:05.689218 17674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:05.689281 17674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:05.689327 17674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:05.698158 17674 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:42387
I20260620 07:17:05.698194 17802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:42387 every 8 connection(s)
I20260620 07:17:05.699002 17674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:17:05.699975 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 17674
I20260620 07:17:05.700101 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:17:05.700413 11758 external_mini_cluster.cc:1428] Setting key 9ff4b888377e95abc2165cfadbc4e2be
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:05.712654 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:05.700917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:39901 (local address 127.11.123.190:33869)
0620 07:17:05.701069 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:05.701088 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:05.702113 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 07:17:05.703511 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:05.703515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:05.703568 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:05.703641 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:05.704750 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.705375 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.706529 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.706759 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.706872 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:05.709276 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:05.709279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:05.709281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:05.709305 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:05.710666 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:05.711161 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:05.711166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:05.711168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:05.711260 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:05.711594 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:05.711599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:05.711601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:05.711780 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:05.711889 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:05.712373 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 07:17:05.712490 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:17:05.713763 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:05.701413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33869 (local address 127.11.123.131:39901)
0620 07:17:05.701974 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:05.702025 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:05.703310 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:05.703816 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:05.703823 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:05.704157 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:05.704602 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:05.704616 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.705496 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.705500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:05.706387 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:05.706396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.706904 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.706908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:05.707084 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:05.707634 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:05.707649 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:05.709050 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:05.710784 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:05.710790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:05.710805 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:05.711056 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:05.711353 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:05.711357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:05.711359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:05.711497 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:05.711886 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:05.711894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:05.712140 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 07:17:05.712333 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 07:17:05.712646 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":147,"threads_started":1}
I20260620 07:17:05.714905 17803 heartbeater.cc:347] Connected to a master server at 127.11.123.190:33869
I20260620 07:17:05.715140 17803 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:05.715624 17803 heartbeater.cc:511] Master 127.11.123.190:33869 requested a full tablet report, sending...
I20260620 07:17:05.716635 17344 ts_manager.cc:194] Registered new tserver with Master: 3d1f242c8ac34ed9a84efc0388de63b9 (127.11.123.131:42387)
I20260620 07:17:05.717320 17344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:39901
I20260620 07:17:05.725953 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/test-admin.keytab.
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:05.742247 11758 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 07:17:05.745031 17810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:05.743564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.11.123.129:44753)
0620 07:17:05.743869 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:05.743885 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:05.743913 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:17:05.743982 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:05.743987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:05.744180 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:05.744338 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:05.744805 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55376: BlockingRecv error: recv got EOF from 127.0.0.1:55376 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
W20260620 07:17:05.745258 11758 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5cc)
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:05.746217 11758 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:05.747260 11758 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.129@KRBTEST.COM
I20260620 07:17:05.756698 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:05.748121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.11.123.129:44753)
0620 07:17:05.748241 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:05.748245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:05.748353 (+ 108us) server_negotiation.cc:407] Connection header received
0620 07:17:05.748471 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:05.748474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:05.748512 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:05.748606 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:05.749167 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.749918 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.750741 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.751132 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.751249 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:05.753516 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:05.753524 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:05.753535 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:05.753555 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:05.755170 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:05.755557 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:05.755561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:05.755563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:05.755641 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:05.755867 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:05.755869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:05.755870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:05.756075 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:05.756222 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:05.756378 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 07:17:05.756537 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 07:17:05.760275 17810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:05.759417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.11.123.129:44753)
0620 07:17:05.759570 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:05.759573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:05.759617 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:17:05.759710 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:05.759712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:05.759745 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:05.759808 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:05.760176 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55396: BlockingRecv error: recv got EOF from 127.0.0.1:55396 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 07:17:05.760376 11758 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5cc' of type FILE
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:05.761358 11758 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:05.762252 11758 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.129@KRBTEST.COM
I20260620 07:17:05.770273 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:05.762986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55410 (local address 127.11.123.129:44753)
0620 07:17:05.763102 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:05.763106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:05.763227 (+ 121us) server_negotiation.cc:407] Connection header received
0620 07:17:05.763360 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:05.763363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:05.763405 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:05.763486 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:05.763941 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.764417 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.765233 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.765449 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.765555 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:05.767603 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:05.767607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:05.767608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:05.767628 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:05.768987 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:05.769387 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:05.769389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:05.769391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:05.769458 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:05.769677 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:05.769679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:05.769680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:05.769816 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:05.769908 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:05.770086 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 07:17:05.770170 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 07:17:05.772689 17810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:05.771817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.11.123.129:44753)
0620 07:17:05.771929 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:05.771933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:05.771998 (+ 65us) server_negotiation.cc:407] Connection header received
0620 07:17:05.772130 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:05.772135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:05.772176 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:05.772259 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:05.772616 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55418: BlockingRecv error: recv got EOF from 127.0.0.1:55418 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 07:17:05.772889 11758 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939794025788-11758-0/krb5kdc/krb5cc)
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:05.773908 11758 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:05.774730 11758 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.129@KRBTEST.COM
I20260620 07:17:05.782908 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:05.775421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.11.123.129:44753)
0620 07:17:05.775528 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:05.775532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:05.775660 (+ 128us) server_negotiation.cc:407] Connection header received
0620 07:17:05.775737 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:05.775739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:05.775769 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:05.775827 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:05.776257 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.776697 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.777517 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:05.777759 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:05.777853 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:05.779916 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:05.779919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:05.779921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:05.779942 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:05.781539 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:05.781897 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:05.781899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:05.781900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:05.781969 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:05.782254 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:05.782257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:05.782258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:05.782400 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:05.782508 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:05.782649 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 07:17:05.782722 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:17:05.783838 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 17406
I20260620 07:17:05.790192 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 17540
I20260620 07:17:05.795902 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 17674
I20260620 07:17:05.800889 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 17312
2026-06-20T07:17:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2187 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17823](info): set up 2 sockets
Jun 20 07:17:05 dist-test-slave-j663 krb5kdc[17823](info): commencing operation
krb5kdc: starting...
W20260620 07:17:06.841459 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:17:06 dist-test-slave-j663 krb5kdc[17823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:17:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.190.keytab.
Entry for principal oryx/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.190.keytab.
I20260620 07:17:06.963510 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:46763
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.190.keytab
--principal=oryx/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:46763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:07.070439 17836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:07.070681 17836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:07.070720 17836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:07.074590 17836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:07.074640 17836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:07.074658 17836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:07.074674 17836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:07.074688 17836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:17:07.079638 17836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:46763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.190.keytab
--principal=oryx/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:46763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:07.080557 17836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:07.081480 17836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:07.089408 17842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:07.089430 17841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:07.089387 17844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:07.089540 17836 server_base.cc:1034] running on GCE node
I20260620 07:17:07.090167 17836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:07.091225 17836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:07.092403 17836 hybrid_clock.cc:648] HybridClock initialized: now 1781939827092392 us; error 28 us; skew 500 ppm
Jun 20 07:17:07 dist-test-slave-j663 krb5kdc[17823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939827, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:07.095106 17836 init.cc:377] Logged in from keytab as oryx/127.11.123.190@KRBTEST.COM (short username oryx)
I20260620 07:17:07.095953 17836 webserver.cc:460] Webserver started at http://127.11.123.190:37255/ using document root <none> and password file <none>
I20260620 07:17:07.096582 17836 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:07.096674 17836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:07.096892 17836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:07.098887 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "20933201ca394c1e97bc08ecdc8d4280"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "70668165e0dd490b9d6274af6d2b0fef"
server_key_iv: "e549d4e1dab3f8c8b991a641193d8729"
server_key_version: "encryptionkey@0"
I20260620 07:17:07.099437 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "20933201ca394c1e97bc08ecdc8d4280"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "70668165e0dd490b9d6274af6d2b0fef"
server_key_iv: "e549d4e1dab3f8c8b991a641193d8729"
server_key_version: "encryptionkey@0"
I20260620 07:17:07.103039 17836 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:17:07.105334 17851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:07.106536 17836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:07.106685 17836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "20933201ca394c1e97bc08ecdc8d4280"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "70668165e0dd490b9d6274af6d2b0fef"
server_key_iv: "e549d4e1dab3f8c8b991a641193d8729"
server_key_version: "encryptionkey@0"
I20260620 07:17:07.106796 17836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:07.124246 17836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:07.127449 17836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:07.127650 17836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:07.134685 17836 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:46763
I20260620 07:17:07.134723 17903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:46763 every 8 connection(s)
I20260620 07:17:07.135607 17836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:07.138345 17904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:07.138826 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 17836
I20260620 07:17:07.138957 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:17:07.139292 11758 external_mini_cluster.cc:1428] Setting key 5a4cab4fcaf76321b7485e85470125c5
I20260620 07:17:07.145324 17904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280: Bootstrap starting.
I20260620 07:17:07.147997 17904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280: Neither blocks nor log segments found. Creating new log.
Jun 20 07:17:07 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:07.148818 17904 log.cc:824] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:07.151135 17904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280: No bootstrap required, opened a new log
I20260620 07:17:07.153673 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:07.140620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.11.123.190:46763)
0620 07:17:07.141000 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:07.141015 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:07.141048 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:17:07.141766 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:07.141784 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:07.142101 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:07.142370 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:07.143239 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.144232 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.145361 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.145860 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.145975 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:07.148608 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:07.148614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:07.148630 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:07.148664 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:07.150652 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:07.151067 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:07.151072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:07.151078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:07.151226 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:07.151502 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:07.151506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:07.151507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:07.151977 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:07.152173 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:07.152364 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 07:17:07.152646 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":133,"threads_started":1}
I20260620 07:17:07.155441 17904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } }
I20260620 07:17:07.155717 17904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:07.155817 17904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20933201ca394c1e97bc08ecdc8d4280, State: Initialized, Role: FOLLOWER
I20260620 07:17:07.156253 17904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } }
I20260620 07:17:07.156404 17904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:07.156471 17904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:07.156566 17904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:07.157495 17904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } }
I20260620 07:17:07.157826 17904 leader_election.cc:302] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20933201ca394c1e97bc08ecdc8d4280; no voters:
I20260620 07:17:07.158116 17904 leader_election.cc:288] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:17:07.158254 17909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:07.158490 17909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 1 LEADER]: Becoming Leader. State: Replica: 20933201ca394c1e97bc08ecdc8d4280, State: Running, Role: LEADER
I20260620 07:17:07.158807 17909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } }
I20260620 07:17:07.159329 17904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:07.160472 17911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20933201ca394c1e97bc08ecdc8d4280. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } }
I20260620 07:17:07.160601 17911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:07.160808 17910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } }
I20260620 07:17:07.160949 17910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:07.160990 17918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:07.164112 17918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:07.169534 17918 catalog_manager.cc:1347] Generated new cluster ID: 8a09ee15963f4a8da59c69cf1b321621
I20260620 07:17:07.169600 17918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:07.179530 17918 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:17:07.180418 17918 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:07.185424 17918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280: Generated new TSK 0
I20260620 07:17:07.186077 17918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab.
Entry for principal oryx/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab.
I20260620 07:17:07.224416 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46763
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab
--principal=oryx/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:07.361886 17930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:07.362215 17930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:07.362289 17930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:07.366358 17930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:07.366443 17930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:07.366536 17930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:17:07.371788 17930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab
--principal=oryx/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:07.372836 17930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:07.373903 17930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:07.383898 17935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:07.383989 17936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:07.384114 17930 server_base.cc:1034] running on GCE node
W20260620 07:17:07.384016 17938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:07.384524 17930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:07.385223 17930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:07.386391 17930 hybrid_clock.cc:648] HybridClock initialized: now 1781939827386375 us; error 28 us; skew 500 ppm
Jun 20 07:17:07 dist-test-slave-j663 krb5kdc[17823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939827, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:07.389302 17930 init.cc:377] Logged in from keytab as oryx/127.11.123.129@KRBTEST.COM (short username oryx)
I20260620 07:17:07.390110 17930 webserver.cc:460] Webserver started at http://127.11.123.129:32931/ using document root <none> and password file <none>
I20260620 07:17:07.390626 17930 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:07.390686 17930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:07.390851 17930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:07.392807 17930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "cbb35bc6d8714b7599cba2fce4b653c2"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "da17bc6c5b99f82dcf3df4006b0d2edb"
server_key_iv: "ec3cc2a7cb748ce64af9dc1fa909f096"
server_key_version: "encryptionkey@0"
I20260620 07:17:07.393405 17930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "cbb35bc6d8714b7599cba2fce4b653c2"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "da17bc6c5b99f82dcf3df4006b0d2edb"
server_key_iv: "ec3cc2a7cb748ce64af9dc1fa909f096"
server_key_version: "encryptionkey@0"
I20260620 07:17:07.397190 17930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:17:07.399662 17945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:07.400981 17930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:17:07.401190 17930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "cbb35bc6d8714b7599cba2fce4b653c2"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "da17bc6c5b99f82dcf3df4006b0d2edb"
server_key_iv: "ec3cc2a7cb748ce64af9dc1fa909f096"
server_key_version: "encryptionkey@0"
I20260620 07:17:07.401315 17930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:07.413622 17930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:07.416730 17930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:07.416939 17930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:07.417608 17930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:07.418395 17930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:07.418465 17930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:07.418534 17930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:07.418587 17930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:07.426918 17930 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:41141
I20260620 07:17:07.426988 18058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:41141 every 8 connection(s)
I20260620 07:17:07.427800 17930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:17:07.431197 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 17930
I20260620 07:17:07.431296 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:17:07.431622 11758 external_mini_cluster.cc:1428] Setting key f03d964671b3d207e517de2a412704f1
Jun 20 07:17:07 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939827, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:17:07.441679 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:07.429608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:42439 (local address 127.11.123.190:46763)
0620 07:17:07.429744 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:07.429748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:07.430731 (+ 983us) server_negotiation.cc:407] Connection header received
0620 07:17:07.432256 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:07.432259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:07.432312 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:07.432396 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:07.433583 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.434107 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.435267 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.435512 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.435624 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:07.438054 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:07.438058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:07.438060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:07.438080 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:07.439482 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:07.440063 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:07.440067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:07.440069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:07.440148 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:07.440489 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:07.440494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:07.440496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:07.440669 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:07.440819 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:07.441349 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 07:17:07.441467 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.130@KRBTEST.COM" created.
I20260620 07:17:07.442560 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:07.430114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.129:42439)
0620 07:17:07.430571 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:07.430613 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:07.432053 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:07.432572 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:07.432579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:07.432909 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:07.433444 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:07.433457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.434242 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.434246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:07.435133 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:07.435142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.435658 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.435663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:07.435832 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:07.436414 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:07.436430 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:07.437889 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:07.439614 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:07.439619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:07.439634 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:07.439962 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:07.440242 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:07.440247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:07.440249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:07.440387 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:07.440794 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:07.440800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:07.441046 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 07:17:07.441296 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 07:17:07.441615 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":113,"threads_started":1}
I20260620 07:17:07.443922 18059 heartbeater.cc:347] Connected to a master server at 127.11.123.190:46763
I20260620 07:17:07.444172 18059 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:07.444669 18059 heartbeater.cc:511] Master 127.11.123.190:46763 requested a full tablet report, sending...
I20260620 07:17:07.446169 17868 ts_manager.cc:194] Registered new tserver with Master: cbb35bc6d8714b7599cba2fce4b653c2 (127.11.123.129:41141)
I20260620 07:17:07.447273 17868 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:42439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab.
Entry for principal oryx/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab.
I20260620 07:17:07.458223 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46763
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab
--principal=oryx/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:07.569881 18064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:07.570137 18064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:07.570178 18064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:07.574086 18064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:07.574136 18064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:07.574191 18064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:17:07.579145 18064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab
--principal=oryx/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:07.580096 18064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:07.581017 18064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:17:07.590389 18064 server_base.cc:1034] running on GCE node
W20260620 07:17:07.590356 18070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:07.590539 18072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:07.590322 18069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:07.590906 18064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:07.591568 18064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:07.592764 18064 hybrid_clock.cc:648] HybridClock initialized: now 1781939827592733 us; error 53 us; skew 500 ppm
Jun 20 07:17:07 dist-test-slave-j663 krb5kdc[17823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939827, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:07.595587 18064 init.cc:377] Logged in from keytab as oryx/127.11.123.130@KRBTEST.COM (short username oryx)
I20260620 07:17:07.596418 18064 webserver.cc:460] Webserver started at http://127.11.123.130:32771/ using document root <none> and password file <none>
I20260620 07:17:07.596962 18064 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:07.597051 18064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:07.597399 18064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:07.599469 18064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "d6fb75ef97ba62563f9c6a5eb87a9932"
server_key_iv: "38af7c98ad3b0c1c42f5182711d9e190"
server_key_version: "encryptionkey@0"
I20260620 07:17:07.599975 18064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "d6fb75ef97ba62563f9c6a5eb87a9932"
server_key_iv: "38af7c98ad3b0c1c42f5182711d9e190"
server_key_version: "encryptionkey@0"
I20260620 07:17:07.603540 18064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 07:17:07.605974 18079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:07.607179 18064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:07.607270 18064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "d6fb75ef97ba62563f9c6a5eb87a9932"
server_key_iv: "38af7c98ad3b0c1c42f5182711d9e190"
server_key_version: "encryptionkey@0"
I20260620 07:17:07.607352 18064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:07.621471 18064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:07.624375 18064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:07.624532 18064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:07.625047 18064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:07.625916 18064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:07.625963 18064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:07.626003 18064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:07.626019 18064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:07.634872 18064 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:42833
I20260620 07:17:07.634896 18192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:42833 every 8 connection(s)
I20260620 07:17:07.635814 18064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:17:07.644449 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 18064
I20260620 07:17:07.644543 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:17:07.644869 11758 external_mini_cluster.cc:1428] Setting key fcd15fc5bd90487c15b640749250b318
Jun 20 07:17:07 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939827, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:07.649456 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:07.637607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:58783 (local address 127.11.123.190:46763)
0620 07:17:07.637774 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:07.637779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:07.638644 (+ 865us) server_negotiation.cc:407] Connection header received
0620 07:17:07.639896 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:07.639899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:07.639959 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:07.640033 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:07.641241 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.641760 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.642859 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.643081 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.643179 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:07.645786 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:07.645790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:07.645792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:07.645816 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:07.647303 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:07.647876 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:07.647881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:07.647885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:07.647986 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:07.648363 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:07.648365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:07.648366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:07.648534 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:07.648609 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:07.649243 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 07:17:07.649338 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:17:07.650698 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:07.638016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.130:58783)
0620 07:17:07.638486 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:07.638526 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:07.639705 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:07.640225 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:07.640232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:07.640577 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:07.641040 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:07.641052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.641862 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.641864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:07.642741 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:07.642750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.643223 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.643225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:07.643361 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:07.643973 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:07.643988 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:07.645646 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:07.647435 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:07.647441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:07.647455 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:07.647742 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:07.648079 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:07.648083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:07.648086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:07.648270 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:07.648633 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:07.648642 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:07.648966 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 07:17:07.649190 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 07:17:07.649535 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
I20260620 07:17:07.652040 18193 heartbeater.cc:347] Connected to a master server at 127.11.123.190:46763
I20260620 07:17:07.652312 18193 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:07.652901 18193 heartbeater.cc:511] Master 127.11.123.190:46763 requested a full tablet report, sending...
I20260620 07:17:07.653962 17868 ts_manager.cc:194] Registered new tserver with Master: 1a5acc88ff0c4c86a67e6b97ea1cee0c (127.11.123.130:42833)
I20260620 07:17:07.654517 17868 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:58783
WARNING: no policy specified for oryx/127.11.123.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab.
Entry for principal oryx/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab.
I20260620 07:17:07.671944 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46763
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab
--principal=oryx/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:07.780110 18198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:07.780376 18198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:07.780421 18198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:07.784482 18198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:07.784543 18198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:07.784651 18198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:17:07.789831 18198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab
--principal=oryx/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:07.790908 18198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:07.791955 18198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:07.801826 18204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:07.801927 18198 server_base.cc:1034] running on GCE node
W20260620 07:17:07.801802 18203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:07.801831 18206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:07.802412 18198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:07.803097 18198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:07.804260 18198 hybrid_clock.cc:648] HybridClock initialized: now 1781939827804244 us; error 28 us; skew 500 ppm
Jun 20 07:17:07 dist-test-slave-j663 krb5kdc[17823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939827, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:07.807224 18198 init.cc:377] Logged in from keytab as oryx/127.11.123.131@KRBTEST.COM (short username oryx)
I20260620 07:17:07.808015 18198 webserver.cc:460] Webserver started at http://127.11.123.131:36145/ using document root <none> and password file <none>
I20260620 07:17:07.808529 18198 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:07.808588 18198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:07.808746 18198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:07.810992 18198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "1dfdc18ed2bc434abea354b7c13b888a"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "50d8cf151e5cb0e473af58149e745d1e"
server_key_iv: "eb5dedb8ccbfd1a0ba44618fb17002e0"
server_key_version: "encryptionkey@0"
I20260620 07:17:07.811626 18198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "1dfdc18ed2bc434abea354b7c13b888a"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "50d8cf151e5cb0e473af58149e745d1e"
server_key_iv: "eb5dedb8ccbfd1a0ba44618fb17002e0"
server_key_version: "encryptionkey@0"
I20260620 07:17:07.815398 18198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 07:17:07.818073 18213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:07.819403 18198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:17:07.819566 18198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "1dfdc18ed2bc434abea354b7c13b888a"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "50d8cf151e5cb0e473af58149e745d1e"
server_key_iv: "eb5dedb8ccbfd1a0ba44618fb17002e0"
server_key_version: "encryptionkey@0"
I20260620 07:17:07.819692 18198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:07.835985 18198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:07.839437 18198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:07.839649 18198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:07.840262 18198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:07.841109 18198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:07.841181 18198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:07.841252 18198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:07.841296 18198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:07.850148 18198 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:36377
I20260620 07:17:07.850170 18326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:36377 every 8 connection(s)
I20260620 07:17:07.851066 18198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:17:07.857910 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 18198
I20260620 07:17:07.858049 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:17:07.858340 11758 external_mini_cluster.cc:1428] Setting key 7af2e53f34769ace5985723eb45e7734
Jun 20 07:17:07 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939827, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:07.864538 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:07.852818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:34493 (local address 127.11.123.190:46763)
0620 07:17:07.853019 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:07.853024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:07.853800 (+ 776us) server_negotiation.cc:407] Connection header received
0620 07:17:07.855092 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:07.855096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:07.855141 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:07.855215 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:07.856349 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.856866 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.858060 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.858383 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.858495 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:07.861374 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:07.861377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:07.861380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:07.861404 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:07.862615 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:07.863104 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:07.863107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:07.863109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:07.863192 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:07.863524 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:07.863527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:07.863529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:07.863703 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:07.863803 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:07.864266 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 07:17:07.864375 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:17:07.865393 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:07.853202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.131:34493)
0620 07:17:07.853644 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:07.853686 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:07.854884 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:07.855402 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:07.855409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:07.855757 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:07.856219 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:07.856232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.856967 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.856970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:07.857910 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:07.857919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:07.858526 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:07.858529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:07.858662 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:07.859294 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:07.859315 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:07.861227 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:07.862735 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:07.862740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:07.862754 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:07.863000 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:07.863282 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:07.863286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:07.863288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:07.863419 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:07.863802 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:07.863808 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:07.864040 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 07:17:07.864214 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 07:17:07.864519 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
I20260620 07:17:07.866503 18327 heartbeater.cc:347] Connected to a master server at 127.11.123.190:46763
I20260620 07:17:07.866720 18327 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:07.867161 18327 heartbeater.cc:511] Master 127.11.123.190:46763 requested a full tablet report, sending...
I20260620 07:17:07.868196 17868 ts_manager.cc:194] Registered new tserver with Master: 1dfdc18ed2bc434abea354b7c13b888a (127.11.123.131:36377)
I20260620 07:17:07.868762 17868 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:34493
I20260620 07:17:07.872087 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.123.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.189.keytab.
Entry for principal oryx/127.11.123.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.189.keytab.
I20260620 07:17:07.900133 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.189:33111
--webserver_interface=127.11.123.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.189.keytab
--principal=oryx/127.11.123.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:46763,127.11.123.189:33111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:08.019707 18332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:08.019994 18332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:08.020035 18332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:08.024210 18332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:08.024274 18332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:08.024294 18332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:08.024310 18332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:08.024326 18332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:17:08.029726 18332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:46763,127.11.123.189:33111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.189.keytab
--principal=oryx/127.11.123.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.189:33111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.123.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
W20260620 07:17:08.029945 18332 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.123.190:46763,127.11.123.189:33111'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 07:17:08.030824 18332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:08.031847 18332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:08.040232 18340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:08.040256 18337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:08.040439 18338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:08.040655 18332 server_base.cc:1034] running on GCE node
I20260620 07:17:08.041260 18332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:08.042358 18332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:08.043558 18332 hybrid_clock.cc:648] HybridClock initialized: now 1781939828043542 us; error 36 us; skew 500 ppm
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:08.046619 18332 init.cc:377] Logged in from keytab as oryx/127.11.123.189@KRBTEST.COM (short username oryx)
I20260620 07:17:08.047472 18332 webserver.cc:460] Webserver started at http://127.11.123.189:46085/ using document root <none> and password file <none>
I20260620 07:17:08.048022 18332 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:08.048120 18332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:08.048343 18332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:08.050433 18332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data/instance:
uuid: "2045f665fa4b4c2499e7ff0cbca5b412"
format_stamp: "Formatted at 2026-06-20 07:17:08 on dist-test-slave-j663"
server_key: "2d9e6e38830288e6f6be6717680b567c"
server_key_iv: "846b5b55060a7c23c77f111d2709a39c"
server_key_version: "encryptionkey@0"
I20260620 07:17:08.050997 18332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/wal/instance:
uuid: "2045f665fa4b4c2499e7ff0cbca5b412"
format_stamp: "Formatted at 2026-06-20 07:17:08 on dist-test-slave-j663"
server_key: "2d9e6e38830288e6f6be6717680b567c"
server_key_iv: "846b5b55060a7c23c77f111d2709a39c"
server_key_version: "encryptionkey@0"
I20260620 07:17:08.054665 18332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:17:08.056954 18347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:08.058152 18332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:08.058295 18332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/wal
uuid: "2045f665fa4b4c2499e7ff0cbca5b412"
format_stamp: "Formatted at 2026-06-20 07:17:08 on dist-test-slave-j663"
server_key: "2d9e6e38830288e6f6be6717680b567c"
server_key_iv: "846b5b55060a7c23c77f111d2709a39c"
server_key_version: "encryptionkey@0"
I20260620 07:17:08.058403 18332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:08.083973 18332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:08.087322 18332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:08.087517 18332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:08.094414 18332 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.189:33111
I20260620 07:17:08.094692 18399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.189:33111 every 8 connection(s)
I20260620 07:17:08.095427 18332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
I20260620 07:17:08.096468 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 18332
I20260620 07:17:08.098693 18400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:08.102382 18400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.123.189@KRBTEST.COM
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.189@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:08.111665 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.097944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.11.123.189:33111)
0620 07:17:08.098438 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.098463 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.098520 (+ 57us) server_negotiation.cc:407] Connection header received
0620 07:17:08.099206 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.099224 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.099559 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.099886 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.100674 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.101735 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.102571 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.103086 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.103250 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.106365 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.106370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.106378 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.106403 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.108246 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.108740 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.108745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.108749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.108856 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.109239 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.109242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.109244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.109758 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.109980 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.110475 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.110851 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":135,"threads_started":1}
I20260620 07:17:08.113690 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.104038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46254 (local address 127.11.123.190:46763)
0620 07:17:08.104185 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.104190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.104594 (+ 404us) server_negotiation.cc:407] Connection header received
0620 07:17:08.104833 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.104835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.104879 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.104952 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.105790 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.106516 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.107410 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.107644 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.107752 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.110268 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.110275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.110278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.110309 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.111999 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.112426 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.112430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.112432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.112528 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.112783 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.112786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.112787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.113007 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.113138 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.113484 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.113582 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:17:08.113752 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.103978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46254)
0620 07:17:08.104438 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.104493 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.104680 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.105033 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.105042 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.105423 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.105642 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.105670 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.106642 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.106645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.107289 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.107297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.107752 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.107756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.107868 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.108348 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.108363 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.110118 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.112107 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.112113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.112116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.112328 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.112619 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.112621 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.112622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.112695 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.113144 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.113164 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.113274 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.113417 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.113581 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":136,"threads_started":1}
I20260620 07:17:08.114832 18400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.189@KRBTEST.COM for oryx/127.11.123.189@KRBTEST.COM
I20260620 07:17:08.122609 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.115374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.11.123.189:33111)
0620 07:17:08.115472 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.115476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.115487 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:17:08.115583 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.115585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.115618 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.115713 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.116057 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.116880 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.117686 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.117921 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.118027 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.119774 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.119777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.119780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.119801 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.121016 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.121477 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.121480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.121482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.121555 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.121822 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.121825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.121827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.121985 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.122085 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.122332 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.122460 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":34}
I20260620 07:17:08.122609 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.115297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:33111 (local address 127.0.0.1:60790)
0620 07:17:08.115408 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.115423 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.115494 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.115722 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.115725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.115879 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.115953 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.115958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.116991 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.116996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.117568 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.117576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.118023 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.118026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.118108 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.118490 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.118500 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.119647 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.121169 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.121172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.121174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.121352 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.121643 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.121648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.121650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.121727 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.122074 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.122078 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.122163 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.122271 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.122408 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 07:17:08.125572 18400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: Bootstrap starting.
I20260620 07:17:08.127888 18400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:08.128587 18400 log.cc:824] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:08.130733 18400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: No bootstrap required, opened a new log
I20260620 07:17:08.133698 18400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } }
I20260620 07:17:08.133899 18400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:08.133935 18400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2045f665fa4b4c2499e7ff0cbca5b412, State: Initialized, Role: FOLLOWER
I20260620 07:17:08.134369 18400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } }
I20260620 07:17:08.134873 18406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } } }
I20260620 07:17:08.135015 18406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:17:08.135541 18400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:17:08.140286 18417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:17:08.140369 18417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:17:08.142212 18332 master_runner.cc:428] Detected that this master 2045f665fa4b4c2499e7ff0cbca5b412 is joining an existing cluster
I20260620 07:17:08.142272 18332 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.123.189:33111
I20260620 07:17:08.156916 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.144430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:33111 (local address 127.0.0.1:60796)
0620 07:17:08.145238 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.145251 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.145364 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.145645 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.145648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.145857 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.145973 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.145980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.146975 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.146978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.147931 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.147940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.148563 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.148566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.148657 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.152861 (+ 4204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.152887 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.153491 (+ 604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.155386 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.155389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.155391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.155593 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.155870 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.155876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.155881 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.155971 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.156357 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.156359 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.156448 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.156544 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.156694 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"spinlock_wait_cycles":3968,"thread_start_us":97,"threads_started":1}
I20260620 07:17:08.157243 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.144926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.11.123.190:46763)
0620 07:17:08.145044 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.145048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.145448 (+ 400us) server_negotiation.cc:407] Connection header received
0620 07:17:08.145620 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.145624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.145682 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.145733 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.146394 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.147255 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.148813 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.149181 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.149260 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.153620 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.153624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.153627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.153658 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.155275 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.155940 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.155943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.155945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.156027 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.156320 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.156326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.156329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.156516 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.156638 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.156957 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.157125 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:17:08.156960 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.144455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.11.123.189:33111)
0620 07:17:08.144585 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.144589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.145309 (+ 720us) server_negotiation.cc:407] Connection header received
0620 07:17:08.145451 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.145453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.145496 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.145538 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.146083 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.146857 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.148045 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.148421 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.148538 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.153627 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.153631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.153634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.153664 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.155265 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.155685 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.155689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.155691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.155770 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.156075 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.156078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.156080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.156257 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.156376 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.156616 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.156758 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":896}
I20260620 07:17:08.157689 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.144998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46258)
0620 07:17:08.145361 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.145374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.145514 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.145888 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.145891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.146127 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.146260 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.146267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.147398 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.147401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.148692 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.148699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.149407 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.149410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.149505 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.152889 (+ 3384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.152901 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.153473 (+ 572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.155675 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.155677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.155678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.155847 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.156130 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.156133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.156135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.156218 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.156680 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.156683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.156778 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.157384 (+ 606us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.157519 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":30,"thread_start_us":126,"threads_started":1}
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939827, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.190@KRBTEST.COM for oryx/127.11.123.189@KRBTEST.COM
I20260620 07:17:08.171321 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.161912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.11.123.189:33111)
0620 07:17:08.162037 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.162043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.162389 (+ 346us) server_negotiation.cc:407] Connection header received
0620 07:17:08.162580 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.162583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.162631 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.162721 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.163469 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.163949 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.164887 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.165195 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.165293 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.167762 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.167765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.167767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.167789 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.169415 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.170013 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.170020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.170021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.170104 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.170390 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.170393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.170394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.170590 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.170701 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.171035 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.171168 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:17:08.171334 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.161905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:33111 (local address 127.0.0.1:60798)
0620 07:17:08.162255 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.162295 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.162439 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.162753 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.162759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.163230 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.163345 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.163351 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.164062 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.164066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.164767 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.164777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.165311 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.165315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.165434 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.165935 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.165955 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.167631 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.169649 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.169659 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.169671 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.169909 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.170199 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.170203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.170205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.170293 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.170701 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.170708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.170831 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.170976 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.171175 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20260620 07:17:08.172806 17868 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.123.189:33111: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412"
member_type: NON_VOTER
last_known_addr {
host: "127.11.123.189"
port: 33111
}
attrs {
promote: true
}
}
dest_uuid: "20933201ca394c1e97bc08ecdc8d4280"
cas_config_opid_index: -1
I20260620 07:17:08.173489 17868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true } }
I20260620 07:17:08.175115 18427 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2045f665fa4b4c2499e7ff0cbca5b412 (127.11.123.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true } } }
I20260620 07:17:08.176009 18374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:08.177342 18427 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.123.189:33111
I20260620 07:17:08.177446 18374 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 FOLLOWER]: Refusing update from remote peer 20933201ca394c1e97bc08ecdc8d4280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 07:17:08.177953 18332 master.cc:537] Master@127.11.123.189:33111 shutting down...
WARNING: no policy specified for oryx/127.11.123.188@KRBTEST.COM; defaulting to no policy
I20260620 07:17:08.178572 18429 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:08.178877 18428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true } } }
I20260620 07:17:08.179097 18428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: This master's current role is: LEADER
W20260620 07:17:08.180694 17854 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.188@KRBTEST.COM" created.
W20260620 07:17:08.182194 17854 consensus_peers.cc:487] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 -> Peer 2045f665fa4b4c2499e7ff0cbca5b412 (127.11.123.189:33111): Couldn't send request to peer 2045f665fa4b4c2499e7ff0cbca5b412. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 07:17:08.182977 18332 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:08.183254 18332 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:08.183355 18332 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: stopping tablet replica
I20260620 07:17:08.187824 18332 master.cc:552] Master@127.11.123.189:33111 shutdown complete.
I20260620 07:17:08.187891 18332 master_runner.cc:305] Clearing existing system tablet
I20260620 07:17:08.188112 18332 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:08.188179 18332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:08.189239 18332 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 07:17:08.190079 18434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:08.190284 18332 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:08.190438 18332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 07:17:08.190474 18332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/wal
uuid: "2045f665fa4b4c2499e7ff0cbca5b412"
format_stamp: "Formatted at 2026-06-20 07:17:08 on dist-test-slave-j663"
server_key: "2d9e6e38830288e6f6be6717680b567c"
server_key_iv: "846b5b55060a7c23c77f111d2709a39c"
server_key_version: "encryptionkey@0"
I20260620 07:17:08.192159 18332 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:08.195621 18332 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 07:17:08.195693 18332 log.cc:1196] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 07:17:08.195993 18332 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: Deleting consensus metadata
I20260620 07:17:08.196162 18332 master_runner.cc:315] Copying system tablet from 127.11.123.190:46763
I20260620 07:17:08.197364 18332 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: tablet copy: Beginning tablet copy session from remote peer at address 127.11.123.190:46763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.188.keytab.
Entry for principal oryx/127.11.123.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.188.keytab.
I20260620 07:17:08.200404 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.188:39431
--webserver_interface=127.11.123.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.188.keytab
--principal=oryx/127.11.123.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:46763,127.11.123.189:33111,127.11.123.188:39431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:17:08.207270 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.198083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.11.123.190:46763)
0620 07:17:08.198422 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.198430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.198575 (+ 145us) server_negotiation.cc:407] Connection header received
0620 07:17:08.198748 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.198751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.198806 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.198904 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.199390 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.200148 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.201398 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.201793 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.201900 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.203818 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.203824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.203829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.203860 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.205579 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.206064 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.206069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.206071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.206155 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.206438 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.206441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.206443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.206631 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.206737 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.206994 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.207101 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260620 07:17:08.207284 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.198073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46272)
0620 07:17:08.198462 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.198477 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.198629 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.198943 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.198947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.199153 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.199263 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.199270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.200272 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.200276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.201259 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.201269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.201907 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.201910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.201995 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.203104 (+ 1109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.203125 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.203631 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.205709 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.205712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.205715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.205941 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.206253 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.206256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.206258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.206344 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.206726 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.206729 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.206831 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.206940 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.207098 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260620 07:17:08.208199 17888 tablet_copy_service.cc:140] P 20933201ca394c1e97bc08ecdc8d4280: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2045f665fa4b4c2499e7ff0cbca5b412 ({username='oryx', principal='oryx/127.11.123.189@KRBTEST.COM'} at 127.0.0.1:46272)
I20260620 07:17:08.208375 17888 tablet_copy_service.cc:161] P 20933201ca394c1e97bc08ecdc8d4280: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2045f665fa4b4c2499e7ff0cbca5b412 at {username='oryx', principal='oryx/127.11.123.189@KRBTEST.COM'} at 127.0.0.1:46272: session id = 2045f665fa4b4c2499e7ff0cbca5b412-00000000000000000000000000000000
I20260620 07:17:08.209981 17888 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 07:17:08.211663 18332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 07:17:08.213652 18332 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: tablet copy: Starting download of 0 data blocks...
I20260620 07:17:08.213853 18332 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: tablet copy: Starting download of 1 WAL segments...
I20260620 07:17:08.215452 18332 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 07:17:08.217010 17888 tablet_copy_service.cc:342] P 20933201ca394c1e97bc08ecdc8d4280: Request end of tablet copy session 2045f665fa4b4c2499e7ff0cbca5b412-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.123.189@KRBTEST.COM'} at 127.0.0.1:46272
I20260620 07:17:08.217190 17888 tablet_copy_service.cc:433] P 20933201ca394c1e97bc08ecdc8d4280: ending tablet copy session 2045f665fa4b4c2499e7ff0cbca5b412-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2045f665fa4b4c2499e7ff0cbca5b412
I20260620 07:17:08.223929 18332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:08.225678 18442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:08.225729 18445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:08.229280 18443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:08.229950 18332 server_base.cc:1034] running on GCE node
I20260620 07:17:08.230120 18332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:08.230422 18332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:08.231564 18332 hybrid_clock.cc:648] HybridClock initialized: now 1781939828231568 us; error 51 us; skew 500 ppm
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:08.232968 18332 init.cc:377] Logged in from keytab as oryx/127.11.123.189@KRBTEST.COM (short username oryx)
I20260620 07:17:08.233511 18332 webserver.cc:460] Webserver started at http://127.11.123.189:41701/ using document root <none> and password file <none>
I20260620 07:17:08.233721 18332 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:08.233775 18332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:08.234958 18332 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 07:17:08.235874 18452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:08.236104 18332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 07:17:08.236215 18332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/wal
uuid: "2045f665fa4b4c2499e7ff0cbca5b412"
format_stamp: "Formatted at 2026-06-20 07:17:08 on dist-test-slave-j663"
server_key: "2d9e6e38830288e6f6be6717680b567c"
server_key_iv: "846b5b55060a7c23c77f111d2709a39c"
server_key_version: "encryptionkey@0"
I20260620 07:17:08.236352 18332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:08.244103 18332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:08.246132 18332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:08.250970 18332 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.189:33111
I20260620 07:17:08.251520 18332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
I20260620 07:17:08.251937 18504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.189:33111 every 8 connection(s)
I20260620 07:17:08.253010 18505 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:17:08.254623 18505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: Bootstrap starting.
I20260620 07:17:08.262874 18505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:17:08.263170 18505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: Bootstrap complete.
I20260620 07:17:08.263645 18505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true } }
I20260620 07:17:08.263777 18505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2045f665fa4b4c2499e7ff0cbca5b412, State: Initialized, Role: LEARNER
I20260620 07:17:08.263887 18505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true } }
I20260620 07:17:08.264219 18508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true } } }
I20260620 07:17:08.264369 18508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:17:08.264389 18505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:08.267509 18519 catalog_manager.cc:1259] Loaded cluster ID: 8a09ee15963f4a8da59c69cf1b321621
I20260620 07:17:08.267599 18519 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: loading cluster ID for follower catalog manager: success
I20260620 07:17:08.269243 18519 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: acquiring CA information for follower catalog manager: success
I20260620 07:17:08.270319 18519 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 07:17:08.352787 18441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:08.353034 18441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:08.353114 18441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:08.356951 18441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:08.357003 18441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:08.357023 18441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:08.357038 18441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:08.357053 18441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:17:08.362406 18441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:46763,127.11.123.189:33111,127.11.123.188:39431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.188.keytab
--principal=oryx/127.11.123.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.188:39431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.123.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:08.363401 18441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:08.364379 18441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:08.372726 18525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:08.372727 18524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:08.372726 18527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:08.372766 18441 server_base.cc:1034] running on GCE node
I20260620 07:17:08.373485 18441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:08.374575 18441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:08.375770 18441 hybrid_clock.cc:648] HybridClock initialized: now 1781939828375755 us; error 42 us; skew 500 ppm
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:08.378443 18441 init.cc:377] Logged in from keytab as oryx/127.11.123.188@KRBTEST.COM (short username oryx)
I20260620 07:17:08.379323 18441 webserver.cc:460] Webserver started at http://127.11.123.188:41081/ using document root <none> and password file <none>
I20260620 07:17:08.379873 18441 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:08.379979 18441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:08.380208 18441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:08.382273 18441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data/instance:
uuid: "d36ebdb692ec4976980191ef158e3e29"
format_stamp: "Formatted at 2026-06-20 07:17:08 on dist-test-slave-j663"
server_key: "f43c9b5ed723fab8e43ba5468a0514f6"
server_key_iv: "7438c05bd2f456330b3d4b5226f98afe"
server_key_version: "encryptionkey@0"
I20260620 07:17:08.382817 18441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/wal/instance:
uuid: "d36ebdb692ec4976980191ef158e3e29"
format_stamp: "Formatted at 2026-06-20 07:17:08 on dist-test-slave-j663"
server_key: "f43c9b5ed723fab8e43ba5468a0514f6"
server_key_iv: "7438c05bd2f456330b3d4b5226f98afe"
server_key_version: "encryptionkey@0"
I20260620 07:17:08.386413 18441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:17:08.388701 18534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:08.389926 18441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:08.390062 18441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/wal
uuid: "d36ebdb692ec4976980191ef158e3e29"
format_stamp: "Formatted at 2026-06-20 07:17:08 on dist-test-slave-j663"
server_key: "f43c9b5ed723fab8e43ba5468a0514f6"
server_key_iv: "7438c05bd2f456330b3d4b5226f98afe"
server_key_version: "encryptionkey@0"
I20260620 07:17:08.390172 18441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:08.413972 18441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:08.416759 18441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:08.416953 18441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:08.424043 18441 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.188:39431
I20260620 07:17:08.424423 18586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.188:39431 every 8 connection(s)
I20260620 07:17:08.425014 18441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
I20260620 07:17:08.427693 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:08.428064 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 18441
I20260620 07:17:08.433406 18587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.123.188@KRBTEST.COM
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.188@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:08.442692 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.429566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.11.123.188:39431)
0620 07:17:08.429945 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.429962 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.430005 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:17:08.430841 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.430864 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.431174 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.431437 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.432318 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.433360 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.434446 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.434951 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.435118 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.437850 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.437855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.437864 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.437889 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.439793 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.440280 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.440284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.440288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.440390 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.440694 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.440699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.440700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.441223 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.441441 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.441711 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.441988 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260620 07:17:08.444592 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.435396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.11.123.190:46763)
0620 07:17:08.435520 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.435524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.435821 (+ 297us) server_negotiation.cc:407] Connection header received
0620 07:17:08.435977 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.435980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.436027 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.436117 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.436909 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.437583 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.438705 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.438925 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.439036 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.441428 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.441431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.441433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.441455 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.442928 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.443388 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.443391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.443393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.443469 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.443753 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.443756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.443758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.443919 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.444028 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.444333 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.444437 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:17:08.444628 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.435229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46286)
0620 07:17:08.435684 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.435733 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.435873 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.436161 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.436167 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.436616 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.436781 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.436788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.437706 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.437709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.438547 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.438556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.439040 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.439043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.439161 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.439697 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.439708 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.441275 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.443049 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.443058 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.443064 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.443277 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.443569 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.443572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.443574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.443656 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.444029 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.444036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.444144 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.444274 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.444438 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":153,"threads_started":1}
I20260620 07:17:08.445667 18587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:17:08.450006 18059 heartbeater.cc:503] Master 127.11.123.190:46763 was elected leader, sending a full tablet report...
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.188@KRBTEST.COM for oryx/127.11.123.189@KRBTEST.COM
I20260620 07:17:08.453619 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.446166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:33111 (local address 127.0.0.1:60804)
0620 07:17:08.446307 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.446323 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.446398 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.446803 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.446806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.446969 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.447061 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.447068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.447937 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.447943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.448461 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.448468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.448918 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.448921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.449015 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.449450 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.449461 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.450770 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.452363 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.452366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.452367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.452539 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.452775 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.452777 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.452778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.452844 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.453262 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.453264 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.453332 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.453411 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.453499 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 07:17:08.453711 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.446365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.11.123.189:33111)
0620 07:17:08.446600 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.446603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.446614 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:17:08.446648 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.446650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.446686 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.446791 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.447178 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.447827 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.448585 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.448808 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.448918 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.450899 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.450901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.450903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.450925 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.452266 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.452627 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.452630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.452631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.452699 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.452914 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.452916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.452917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.453183 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.453267 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.453466 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.453560 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260620 07:17:08.454365 18587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.188@KRBTEST.COM for oryx/127.11.123.188@KRBTEST.COM
I20260620 07:17:08.461709 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.454781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:39431 (local address 127.0.0.1:35636)
0620 07:17:08.454885 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.454900 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.455018 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.455203 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.455205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.455321 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.455382 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.455394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.456386 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.456388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.456859 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.456866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.457375 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.457378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.457437 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.457773 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.457784 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.458823 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.460426 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.460429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.460430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.460677 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.460890 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.460891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.460893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.460960 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.461307 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.461309 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.461429 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.461507 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.461591 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 07:17:08.461735 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.454856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.11.123.188:39431)
0620 07:17:08.454954 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.454958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.454987 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:17:08.455101 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.455103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.455135 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.455189 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.455475 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.455929 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.456972 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.457281 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.457360 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.458934 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.458937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.458939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.458958 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.460329 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.460745 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.460747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.460748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.460813 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.461009 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.461011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.461012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.461227 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.461300 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.461559 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.461637 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 07:17:08.464447 18587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: Bootstrap starting.
I20260620 07:17:08.466945 18587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:08.467648 18587 log.cc:824] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:08.469425 18587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: No bootstrap required, opened a new log
I20260620 07:17:08.472204 18587 raft_consensus.cc:348] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } }
I20260620 07:17:08.472394 18587 raft_consensus.cc:374] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:08.472471 18587 raft_consensus.cc:729] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36ebdb692ec4976980191ef158e3e29, State: Initialized, Role: FOLLOWER
I20260620 07:17:08.472873 18587 consensus_queue.cc:260] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } }
I20260620 07:17:08.473325 18593 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } } }
I20260620 07:17:08.473461 18593 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:17:08.474001 18587 sys_catalog.cc:564] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:17:08.478602 18604 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:17:08.478699 18604 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:17:08.482833 18441 master_runner.cc:428] Detected that this master d36ebdb692ec4976980191ef158e3e29 is joining an existing cluster
I20260620 07:17:08.482913 18441 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.123.188:39431
I20260620 07:17:08.494241 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.484880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46300)
0620 07:17:08.485423 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.485437 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.485605 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.486021 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.486027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.486402 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.486701 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.486707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.487758 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.487762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.489342 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.489348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.490041 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.490043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.490107 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.490548 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.490569 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.490907 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.492639 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.492642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.492652 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.492905 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.493245 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.493249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.493251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.493340 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.493737 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.493740 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.493846 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.493957 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.494097 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":105,"thread_start_us":89,"threads_started":1}
I20260620 07:17:08.494341 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.484948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.11.123.190:46763)
0620 07:17:08.485096 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.485101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.485511 (+ 410us) server_negotiation.cc:407] Connection header received
0620 07:17:08.485749 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.485753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.485815 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.485865 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.486830 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.487616 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.489462 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.489819 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.489898 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.491046 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.491050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.491052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.491077 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.492515 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.493007 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.493012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.493014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.493141 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.493438 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.493456 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.493459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.493638 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.493750 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.494069 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.494185 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:17:08.495651 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 17930
I20260620 07:17:08.504313 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:41141
--local_ip_for_outbound_sockets=127.11.123.129
--tserver_master_addrs=127.11.123.190:46763
--webserver_port=32931
--webserver_interface=127.11.123.129
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab
--principal=oryx/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.123.190:46763,127.11.123.189:33111
--tserver_master_addrs=127.11.123.190:46763,127.11.123.189:33111,127.11.123.188:39431 with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939827, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.190@KRBTEST.COM for oryx/127.11.123.188@KRBTEST.COM
I20260620 07:17:08.514067 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.485104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:33111 (local address 127.0.0.1:60812)
0620 07:17:08.485610 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.485622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.485740 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.486085 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.486088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.486460 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.486694 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.486702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.487758 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.487762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.488775 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.488784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.489494 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.489497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.489581 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.509600 (+ 20019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.509621 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.510179 (+ 558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.512213 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.512217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.512219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.512539 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.512836 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.512839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.512841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.512920 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.513388 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.513391 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.513478 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.513572 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.513726 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":80,"thread_start_us":87,"threads_started":1}
I20260620 07:17:08.514885 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.484953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.11.123.189:33111)
0620 07:17:08.485220 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.485224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.485686 (+ 462us) server_negotiation.cc:407] Connection header received
0620 07:17:08.485841 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.485846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.485894 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.485950 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.486829 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.487616 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.488906 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.489279 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.489349 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.510331 (+ 20982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.510337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.510342 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.510374 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.512053 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.512654 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.512657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.512659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.512743 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.513013 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.513016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.513017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.513260 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.513347 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.514597 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.514734 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260620 07:17:08.515950 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.499147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:39431 (local address 127.0.0.1:35658)
0620 07:17:08.499259 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.499274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.499349 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.501854 (+ 2505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.501857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.502056 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.502155 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.502162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.503325 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.503329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.504185 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.504195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.504835 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.504839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.504936 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.505471 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.505485 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.506997 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.514016 (+ 7019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.514022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.514024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.514298 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.514678 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.514684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.514687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.514789 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.515298 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.515304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.515424 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.515522 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.515709 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 07:17:08.516661 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.501227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.11.123.188:39431)
0620 07:17:08.501532 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.501542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.501559 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:17:08.501640 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.501653 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.501717 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.501814 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.502284 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.503211 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.504313 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.504702 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.504819 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.507213 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.507217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.507220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.507244 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.513873 (+ 6629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.514441 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.514445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.514448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.514543 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.514922 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.514927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.514930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.515168 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.516246 (+ 1078us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.516320 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.516489 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":104,"threads_started":1}
I20260620 07:17:08.517529 17868 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.123.188:39431: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d36ebdb692ec4976980191ef158e3e29"
member_type: NON_VOTER
last_known_addr {
host: "127.11.123.188"
port: 39431
}
attrs {
promote: true
}
}
dest_uuid: "20933201ca394c1e97bc08ecdc8d4280"
cas_config_opid_index: 5
I20260620 07:17:08.518141 17868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } }
I20260620 07:17:08.518808 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.485364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.11.123.188:39431)
0620 07:17:08.485450 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.485457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.485734 (+ 277us) server_negotiation.cc:407] Connection header received
0620 07:17:08.485880 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.485882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.485918 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.485974 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.486806 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.487550 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.488686 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.489053 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.489176 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.509899 (+ 20723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.509903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.509906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.509930 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.512891 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.516634 (+ 3743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.516637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.516639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.516705 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.517591 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.517594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.517596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.517919 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.518000 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.518567 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.518667 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":32}
I20260620 07:17:08.519388 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.485328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:39431 (local address 127.0.0.1:35646)
0620 07:17:08.485657 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.485671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.485786 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.486107 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.486110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.486460 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.486685 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.486694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.487665 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.487668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.488568 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.488578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.489185 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.489189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.489307 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.489929 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.489944 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.509417 (+ 19473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.516218 (+ 6801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.516222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.516224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.516535 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.517403 (+ 868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.517407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.517409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.517499 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.518239 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.518242 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.518398 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.519090 (+ 692us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.519224 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":79,"thread_start_us":80,"threads_started":1}
I20260620 07:17:08.519423 18428 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d36ebdb692ec4976980191ef158e3e29 (127.11.123.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:08.520816 18428 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.123.188:39431
I20260620 07:17:08.521004 18428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:08.521128 18428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:08.522056 18441 master.cc:537] Master@127.11.123.188:39431 shutting down...
I20260620 07:17:08.522166 18561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:08.523614 18561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 FOLLOWER]: Refusing update from remote peer 20933201ca394c1e97bc08ecdc8d4280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 07:17:08.524216 18429 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:17:08.527223 17855 consensus_peers.cc:487] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 -> Peer d36ebdb692ec4976980191ef158e3e29 (127.11.123.188:39431): Couldn't send request to peer d36ebdb692ec4976980191ef158e3e29. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 07:17:08.528753 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.521265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:33111 (local address 127.0.0.1:60826)
0620 07:17:08.521386 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.521401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.521474 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.524514 (+ 3040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.524518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.524537 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:08.524628 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.524634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.525918 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.525922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.527218 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.527229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.528331 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.528334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.528427 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.528488 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.528550 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.528623 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 07:17:08.529361 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.521612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.11.123.189:33111)
0620 07:17:08.522924 (+ 1312us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.522929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.522942 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:17:08.522982 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.522985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.523040 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.523150 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:08.524753 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.525790 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.527435 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.528115 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.528970 (+ 855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.529150 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.529186 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.529248 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237}
I20260620 07:17:08.529398 18441 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:08.529650 18441 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:08.529738 18441 tablet_replica.cc:330] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: stopping tablet replica
I20260620 07:17:08.529832 18479 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 LEARNER]: Refusing update from remote peer 20933201ca394c1e97bc08ecdc8d4280: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 07:17:08.530524 18429 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 07:17:08.532260 18479 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d36ebdb692ec4976980191ef158e3e29 (127.11.123.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:08.534549 18508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20933201ca394c1e97bc08ecdc8d4280. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:08.534672 18508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:17:08.534754 18508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: NON_VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: true } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:08.534823 18508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:17:08.536152 18429 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280: attempting to promote NON_VOTER 2045f665fa4b4c2499e7ff0cbca5b412 to VOTER
I20260620 07:17:08.536378 18441 master.cc:552] Master@127.11.123.188:39431 shutdown complete.
I20260620 07:17:08.536455 18441 master_runner.cc:305] Clearing existing system tablet
I20260620 07:17:08.536554 18429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } }
I20260620 07:17:08.536809 18441 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:08.536895 18441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:08.537863 18479 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 LEARNER]: Refusing update from remote peer 20933201ca394c1e97bc08ecdc8d4280: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 07:17:08.538206 18429 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 07:17:08.539276 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.537459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:39431 (local address 127.0.0.1:35668)
0620 07:17:08.539138 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.539208 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1554}
I20260620 07:17:08.540102 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.539680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:39431 (local address 127.0.0.1:35670)
0620 07:17:08.539970 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.540023 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
W20260620 07:17:08.540267 17855 consensus_peers.cc:487] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 -> Peer d36ebdb692ec4976980191ef158e3e29 (127.11.123.188:39431): Couldn't send request to peer d36ebdb692ec4976980191ef158e3e29. Status: Network error: Client connection negotiation failed: client connection to 127.11.123.188:39431: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 07:17:08.540571 18429 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2045f665fa4b4c2499e7ff0cbca5b412 (127.11.123.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:08.541456 18441 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 07:17:08.541504 18478 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2045f665fa4b4c2499e7ff0cbca5b412 (127.11.123.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:08.542256 18429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:08.542356 18429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:08.542474 18429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:08.542560 18429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:08.542976 18616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:08.543083 18616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:17:08.543325 18619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:08.543524 18441 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:08.543653 18441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260620 07:17:08.543686 18441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/wal
uuid: "d36ebdb692ec4976980191ef158e3e29"
format_stamp: "Formatted at 2026-06-20 07:17:08 on dist-test-slave-j663"
server_key: "f43c9b5ed723fab8e43ba5468a0514f6"
server_key_iv: "7438c05bd2f456330b3d4b5226f98afe"
server_key_version: "encryptionkey@0"
I20260620 07:17:08.545681 18441 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:08.547890 18441 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 07:17:08.547977 18441 log.cc:1196] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 07:17:08.548211 18441 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: Deleting consensus metadata
I20260620 07:17:08.548372 18441 master_runner.cc:315] Copying system tablet from 127.11.123.190:46763
I20260620 07:17:08.549512 18441 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: tablet copy: Beginning tablet copy session from remote peer at address 127.11.123.190:46763
I20260620 07:17:08.561771 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.550042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.11.123.190:46763)
0620 07:17:08.550223 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.550227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.550422 (+ 195us) server_negotiation.cc:407] Connection header received
0620 07:17:08.550546 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.550552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.550611 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.550710 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.551189 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.551966 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.553112 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.553446 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.553560 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.557883 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.557886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.557889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.557914 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.559661 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.560221 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.560224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.560226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.560315 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.560706 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.560711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.560714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.560970 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.561114 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.561499 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.561616 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 07:17:08.561944 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.549967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46314)
0620 07:17:08.550353 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.550365 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.550462 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.550737 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.550740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.550954 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.551068 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.551075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.552088 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.552092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.552957 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.552966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.553571 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.553575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.553661 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.557266 (+ 3605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.557281 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.557708 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.559845 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.559848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.559851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.560109 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.560423 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.560426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.560428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.560588 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.561171 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.561177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.561302 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.561436 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.561678 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":89,"threads_started":1}
I20260620 07:17:08.562525 17888 tablet_copy_service.cc:140] P 20933201ca394c1e97bc08ecdc8d4280: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d36ebdb692ec4976980191ef158e3e29 ({username='oryx', principal='oryx/127.11.123.188@KRBTEST.COM'} at 127.0.0.1:46314)
I20260620 07:17:08.562634 17888 tablet_copy_service.cc:161] P 20933201ca394c1e97bc08ecdc8d4280: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d36ebdb692ec4976980191ef158e3e29 at {username='oryx', principal='oryx/127.11.123.188@KRBTEST.COM'} at 127.0.0.1:46314: session id = d36ebdb692ec4976980191ef158e3e29-00000000000000000000000000000000
I20260620 07:17:08.563648 17888 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 07:17:08.564980 18441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 07:17:08.567323 18441 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: tablet copy: Starting download of 0 data blocks...
I20260620 07:17:08.567567 18441 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: tablet copy: Starting download of 1 WAL segments...
I20260620 07:17:08.569051 18441 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 07:17:08.570750 17888 tablet_copy_service.cc:342] P 20933201ca394c1e97bc08ecdc8d4280: Request end of tablet copy session d36ebdb692ec4976980191ef158e3e29-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.123.188@KRBTEST.COM'} at 127.0.0.1:46314
I20260620 07:17:08.570832 17888 tablet_copy_service.cc:433] P 20933201ca394c1e97bc08ecdc8d4280: ending tablet copy session d36ebdb692ec4976980191ef158e3e29-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d36ebdb692ec4976980191ef158e3e29
I20260620 07:17:08.577730 18441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:08.579407 18626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:08.579444 18627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:08.579653 18629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:08.581261 18441 server_base.cc:1034] running on GCE node
I20260620 07:17:08.581414 18441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:08.581691 18441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:08.582815 18441 hybrid_clock.cc:648] HybridClock initialized: now 1781939828582802 us; error 27 us; skew 500 ppm
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:08.584439 18441 init.cc:377] Logged in from keytab as oryx/127.11.123.188@KRBTEST.COM (short username oryx)
I20260620 07:17:08.584877 18441 webserver.cc:460] Webserver started at http://127.11.123.188:35191/ using document root <none> and password file <none>
I20260620 07:17:08.585067 18441 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:08.585160 18441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:08.586390 18441 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 07:17:08.587366 18636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:08.587594 18441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 07:17:08.587661 18441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/wal
uuid: "d36ebdb692ec4976980191ef158e3e29"
format_stamp: "Formatted at 2026-06-20 07:17:08 on dist-test-slave-j663"
server_key: "f43c9b5ed723fab8e43ba5468a0514f6"
server_key_iv: "7438c05bd2f456330b3d4b5226f98afe"
server_key_version: "encryptionkey@0"
I20260620 07:17:08.587742 18441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:08.623533 18441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:08.625830 18441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:08.630810 18441 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.188:39431
I20260620 07:17:08.630951 18688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.188:39431 every 8 connection(s)
I20260620 07:17:08.631381 18441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
I20260620 07:17:08.633114 18689 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:17:08.634100 18689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: Bootstrap starting.
I20260620 07:17:08.645180 18689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:17:08.645463 18689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: Bootstrap complete.
I20260620 07:17:08.645999 18689 raft_consensus.cc:348] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } }
I20260620 07:17:08.646134 18689 raft_consensus.cc:729] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d36ebdb692ec4976980191ef158e3e29, State: Initialized, Role: LEARNER
I20260620 07:17:08.646237 18689 consensus_queue.cc:260] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } }
I20260620 07:17:08.646710 18695 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:08.646853 18689 sys_catalog.cc:564] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:08.646979 18695 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:17:08.650386 18706 catalog_manager.cc:1259] Loaded cluster ID: 8a09ee15963f4a8da59c69cf1b321621
I20260620 07:17:08.650440 18706 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: loading cluster ID for follower catalog manager: success
W20260620 07:17:08.650746 18614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:08.651065 18614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:08.651139 18614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 07:17:08.652537 18706 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: acquiring CA information for follower catalog manager: success
I20260620 07:17:08.653836 18706 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 07:17:08.655972 18614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:08.656023 18614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:08.656121 18614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:17:08.656476 18193 heartbeater.cc:503] Master 127.11.123.190:46763 was elected leader, sending a full tablet report...
I20260620 07:17:08.661945 18614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab
--principal=oryx/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:41141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=32931
--tserver_master_addrs=127.11.123.190:46763,127.11.123.189:33111,127.11.123.188:39431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:08.662814 18614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:08.663739 18614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:08.673007 18711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:08.673033 18709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:08.673051 18708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:08.673323 18614 server_base.cc:1034] running on GCE node
I20260620 07:17:08.673717 18614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:08.674340 18614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:08.675546 18614 hybrid_clock.cc:648] HybridClock initialized: now 1781939828675514 us; error 47 us; skew 500 ppm
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:08.678552 18614 init.cc:377] Logged in from keytab as oryx/127.11.123.129@KRBTEST.COM (short username oryx)
I20260620 07:17:08.679431 18614 webserver.cc:460] Webserver started at http://127.11.123.129:32931/ using document root <none> and password file <none>
I20260620 07:17:08.680083 18614 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:08.680204 18614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:08.684165 18614 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:08.686290 18718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:08.687601 18614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:08.687760 18614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "cbb35bc6d8714b7599cba2fce4b653c2"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "da17bc6c5b99f82dcf3df4006b0d2edb"
server_key_iv: "ec3cc2a7cb748ce64af9dc1fa909f096"
server_key_version: "encryptionkey@0"
I20260620 07:17:08.688333 18614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:08.706431 18614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:08.709182 18614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:08.709380 18614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:08.709928 18614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:08.710888 18614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:08.710963 18614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:08.711059 18614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:08.711112 18614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:08.719348 18614 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:41141
I20260620 07:17:08.719417 18831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:41141 every 8 connection(s)
I20260620 07:17:08.720359 18614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:17:08.723778 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 18614
I20260620 07:17:08.723990 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 18064
I20260620 07:17:08.731280 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:42833
--local_ip_for_outbound_sockets=127.11.123.130
--tserver_master_addrs=127.11.123.190:46763
--webserver_port=32771
--webserver_interface=127.11.123.130
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab
--principal=oryx/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.123.190:46763,127.11.123.189:33111
--tserver_master_addrs=127.11.123.190:46763,127.11.123.189:33111,127.11.123.188:39431 with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.189@KRBTEST.COM
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.188@KRBTEST.COM
I20260620 07:17:08.738672 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.722240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:41347 (local address 127.11.123.190:46763)
0620 07:17:08.722385 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.722390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.723249 (+ 859us) server_negotiation.cc:407] Connection header received
0620 07:17:08.725348 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.725354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.725409 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.725472 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.727657 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.728400 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.730254 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.730600 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.730678 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.734025 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.734029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.734031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.734054 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.735808 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.736478 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.736482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.736485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.736583 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.737030 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.737035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.737038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.737297 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.737401 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.738368 (+ 967us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.738484 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:17:08.739364 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.722140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:39843 (local address 127.11.123.189:33111)
0620 07:17:08.722271 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.722275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.723262 (+ 987us) server_negotiation.cc:407] Connection header received
0620 07:17:08.725185 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.725192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.725261 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.725321 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.727148 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.728057 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.730261 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.730620 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.730708 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.735225 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.735230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.735233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.735264 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.737489 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.737960 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.737964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.737966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.738045 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.738337 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.738340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.738342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.738554 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.738633 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.739105 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.739223 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:17:08.739542 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.722594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:33111 (local address 127.11.123.129:39843)
0620 07:17:08.723110 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.723157 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.724925 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.725648 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.725657 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.726303 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.726964 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.726985 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.728236 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.728242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.730121 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.730135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.730889 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.730893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.731180 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.732163 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.732184 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.735088 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.737610 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.737613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.737615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.737846 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.738148 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.738150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.738152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.738231 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.738693 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.738695 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.738908 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.739030 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.739187 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":332,"spinlock_wait_cycles":16256,"thread_start_us":142,"threads_started":1}
I20260620 07:17:08.739710 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.722594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.129:41347)
0620 07:17:08.723109 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.723156 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.724914 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.725647 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.725655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.726603 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.727523 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.727532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.728540 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.728543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.730121 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.730131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.730834 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.730838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.731152 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.731996 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.732011 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.733848 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.735952 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.735960 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.735976 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.736353 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.736698 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.736704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.736708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.736897 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.737458 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.737464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.737764 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.737949 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.738330 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":1417,"spinlock_wait_cycles":8064,"thread_start_us":194,"threads_started":1}
I20260620 07:17:08.740787 18834 heartbeater.cc:347] Connected to a master server at 127.11.123.189:33111
I20260620 07:17:08.741024 18834 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:08.741235 18833 heartbeater.cc:347] Connected to a master server at 127.11.123.190:46763
I20260620 07:17:08.741303 18833 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:08.741582 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.723603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:39431 (local address 127.11.123.129:41305)
0620 07:17:08.723930 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.723946 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.725131 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.725647 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.725655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.726245 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.726964 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.726986 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.728368 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.728371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.730121 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.730131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.730834 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.730838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.731172 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.731837 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.731853 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.735475 (+ 3622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.738177 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.738183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.738186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.740248 (+ 2062us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.740557 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.740560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.740562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.740653 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.741132 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.741135 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.741222 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.741325 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.741455 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":428,"spinlock_wait_cycles":9344,"thread_start_us":80,"threads_started":1}
I20260620 07:17:08.741720 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.723988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:41305 (local address 127.11.123.188:39431)
0620 07:17:08.725194 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.725198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.725212 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:17:08.725254 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.725257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.725311 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.725393 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.727133 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.728249 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.730254 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.730607 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.730689 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.735637 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.735644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.735650 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.735682 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.738041 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.740362 (+ 2321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.740366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.740368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.740453 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.740766 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.740769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.740771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.741000 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.741137 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.741424 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.741550 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114,"thread_start_us":81,"threads_started":1}
I20260620 07:17:08.741922 18833 heartbeater.cc:511] Master 127.11.123.190:46763 requested a full tablet report, sending...
I20260620 07:17:08.742388 18832 heartbeater.cc:347] Connected to a master server at 127.11.123.188:39431
I20260620 07:17:08.742470 18832 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:08.742606 18832 heartbeater.cc:511] Master 127.11.123.188:39431 requested a full tablet report, sending...
I20260620 07:17:08.742731 18834 heartbeater.cc:511] Master 127.11.123.189:33111 requested a full tablet report, sending...
I20260620 07:17:08.743177 17868 ts_manager.cc:194] Re-registered known tserver with Master: cbb35bc6d8714b7599cba2fce4b653c2 (127.11.123.129:41141)
I20260620 07:17:08.743686 18645 ts_manager.cc:194] Registered new tserver with Master: cbb35bc6d8714b7599cba2fce4b653c2 (127.11.123.129:41141)
I20260620 07:17:08.743686 18461 ts_manager.cc:194] Registered new tserver with Master: cbb35bc6d8714b7599cba2fce4b653c2 (127.11.123.129:41141)
I20260620 07:17:08.744036 17868 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:41347
W20260620 07:17:08.847509 18840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:08.847761 18840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:08.847841 18840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:08.852416 18840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:08.852486 18840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:08.852572 18840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:17:08.857664 18840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab
--principal=oryx/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:42833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=32771
--tserver_master_addrs=127.11.123.190:46763,127.11.123.189:33111,127.11.123.188:39431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:08.858615 18840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:08.859635 18840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:08.869302 18846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:08.869323 18848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:08.869316 18845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:08.869825 18840 server_base.cc:1034] running on GCE node
I20260620 07:17:08.870203 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:08.870682 18327 heartbeater.cc:503] Master 127.11.123.190:46763 was elected leader, sending a full tablet report...
I20260620 07:17:08.870883 18840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:08.872049 18840 hybrid_clock.cc:648] HybridClock initialized: now 1781939828872030 us; error 30 us; skew 500 ppm
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:08.874675 18840 init.cc:377] Logged in from keytab as oryx/127.11.123.130@KRBTEST.COM (short username oryx)
I20260620 07:17:08.875512 18840 webserver.cc:460] Webserver started at http://127.11.123.130:32771/ using document root <none> and password file <none>
I20260620 07:17:08.876045 18840 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:08.876148 18840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:08.879820 18840 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:08.881880 18855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:08.883013 18840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:08.883145 18840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "d6fb75ef97ba62563f9c6a5eb87a9932"
server_key_iv: "38af7c98ad3b0c1c42f5182711d9e190"
server_key_version: "encryptionkey@0"
I20260620 07:17:08.883657 18840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:08.901558 18840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:08.904207 18840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:08.904399 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:08.904955 18840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:08.905825 18840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:08.905870 18840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:08.905941 18840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:08.905972 18840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:08.915067 18840 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:42833
I20260620 07:17:08.915117 18968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:42833 every 8 connection(s)
I20260620 07:17:08.916023 18840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:17:08.918852 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 18840
I20260620 07:17:08.919032 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 18198
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.188@KRBTEST.COM
I20260620 07:17:08.929188 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:36377
--local_ip_for_outbound_sockets=127.11.123.131
--tserver_master_addrs=127.11.123.190:46763
--webserver_port=36145
--webserver_interface=127.11.123.131
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab
--principal=oryx/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.123.190:46763,127.11.123.189:33111
--tserver_master_addrs=127.11.123.190:46763,127.11.123.189:33111,127.11.123.188:39431 with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:08.931536 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.926547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35684 (local address 127.11.123.188:39431)
0620 07:17:08.926924 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.926930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.926946 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:17:08.926992 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.926997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.927064 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.927855 (+ 791us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:08.927973 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.928755 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.930260 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.930909 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.931054 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.931203 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.931326 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.931396 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":100,"threads_started":1}
I20260620 07:17:08.931586 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.926062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:39431 (local address 127.0.0.1:35684)
0620 07:17:08.926232 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.926248 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.926341 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.927581 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.927587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.927614 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:08.927736 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.927746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.928906 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.928909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.930110 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.930130 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.931057 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.931061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.931188 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.931217 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.931315 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.931406 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 07:17:08.932160 18655 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 07:17:08.932533 18695 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20933201ca394c1e97bc08ecdc8d4280. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:08.932680 18695 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:17:08.934765 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.921278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:42447 (local address 127.11.123.188:39431)
0620 07:17:08.921412 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.921418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.921433 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:17:08.921479 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.921482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.921532 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.921621 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.923450 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.924266 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.925943 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.926310 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.926383 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.930480 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.930486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.930491 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.930523 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.932204 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.932806 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.932810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.932812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.932885 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.933380 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.933386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.933389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.933619 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.933698 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.934418 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.934555 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 07:17:08 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939828, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.189@KRBTEST.COM
I20260620 07:17:08.936439 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.918537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:39431 (local address 127.11.123.130:42447)
0620 07:17:08.919243 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.919355 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.920756 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.921934 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.921946 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.922537 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.923266 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.923289 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.924423 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.924426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.925812 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.925820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.926582 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.926586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.926785 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.927470 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.927492 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.930277 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.932320 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.932326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.932342 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.932703 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.933008 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.933013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.933019 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.933258 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.933788 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.933796 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.934102 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.934909 (+ 807us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.935286 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":144,"threads_started":1}
I20260620 07:17:08.936594 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.921547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:59421 (local address 127.11.123.190:46763)
0620 07:17:08.921669 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.921674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.923763 (+ 2089us) server_negotiation.cc:407] Connection header received
0620 07:17:08.923928 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.923931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.923976 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.924024 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.924654 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.925449 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.926561 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.926879 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.926961 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.932758 (+ 5797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.932764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.932769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.932797 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.934639 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.935143 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.935147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.935149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.935233 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.935567 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.935570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.935572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.935787 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.935896 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.936351 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.936458 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 07:17:08.936859 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.919630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.130:59421)
0620 07:17:08.923676 (+ 4046us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.923691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.923826 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.924154 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.924156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.924374 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.924506 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.924512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.925588 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.925591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.926437 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.926445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.927122 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.927126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.927239 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.927743 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.927758 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.932592 (+ 4834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.934761 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.934765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.934767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.935022 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.935350 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.935356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.935361 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.935455 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.935973 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.935976 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.936084 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.936195 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.936339 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3967,"thread_start_us":76,"threads_started":1}
I20260620 07:17:08.937973 18969 heartbeater.cc:347] Connected to a master server at 127.11.123.188:39431
I20260620 07:17:08.938179 18970 heartbeater.cc:347] Connected to a master server at 127.11.123.190:46763
I20260620 07:17:08.938321 18970 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:08.938444 18969 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:08.938905 18970 heartbeater.cc:511] Master 127.11.123.190:46763 requested a full tablet report, sending...
I20260620 07:17:08.938905 18969 heartbeater.cc:511] Master 127.11.123.188:39431 requested a full tablet report, sending...
I20260620 07:17:08.940083 17868 ts_manager.cc:194] Re-registered known tserver with Master: 1a5acc88ff0c4c86a67e6b97ea1cee0c (127.11.123.130:42833)
I20260620 07:17:08.940092 18645 ts_manager.cc:194] Registered new tserver with Master: 1a5acc88ff0c4c86a67e6b97ea1cee0c (127.11.123.130:42833)
I20260620 07:17:08.940721 17868 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:59421
I20260620 07:17:08.941936 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.921170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:33473 (local address 127.11.123.189:33111)
0620 07:17:08.922412 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:08.922416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:08.928904 (+ 6488us) server_negotiation.cc:407] Connection header received
0620 07:17:08.929074 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:08.929113 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:08.929163 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:08.929217 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:08.930059 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.930803 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.933226 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.933592 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.933696 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.937861 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:08.937866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:08.937873 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:08.937897 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:08.939773 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.940397 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.940403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.940406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.940505 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:08.940852 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:08.940857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:08.940860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:08.941132 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:08.941241 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:08.941660 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 07:17:08.941775 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155}
I20260620 07:17:08.942023 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:08.918537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:33111 (local address 127.11.123.130:33473)
0620 07:17:08.928789 (+ 10252us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:08.928805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:08.928971 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:08.929351 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:08.929354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:08.929816 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:08.929931 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.929938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.930953 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.930959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.933066 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:08.933079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:08.933888 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:08.933895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:08.934005 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:08.934484 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:08.934501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:08.937735 (+ 3234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:08.939935 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.939941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.939946 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.940267 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.940629 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:08.940635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:08.940637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:08.940734 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:08.941307 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:08.941313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:08.941429 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 07:17:08.941546 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 07:17:08.941754 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10158,"mutex_wait_us":1412,"thread_start_us":118,"threads_started":1}
I20260620 07:17:08.942672 18971 heartbeater.cc:347] Connected to a master server at 127.11.123.189:33111
I20260620 07:17:08.942747 18971 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:08.942912 18971 heartbeater.cc:511] Master 127.11.123.189:33111 requested a full tablet report, sending...
I20260620 07:17:08.943686 18461 ts_manager.cc:194] Registered new tserver with Master: 1a5acc88ff0c4c86a67e6b97ea1cee0c (127.11.123.130:42833)
W20260620 07:17:09.044901 18977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:09.045217 18977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:09.045264 18977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:09.049516 18977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:09.049566 18977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:09.049624 18977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:17:09.054627 18977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab
--principal=oryx/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:36377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=36145
--tserver_master_addrs=127.11.123.190:46763,127.11.123.189:33111,127.11.123.188:39431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:09.055552 18977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:09.056465 18977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:09.066286 18983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:09.066273 18985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:09.066320 18982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:09.066689 18977 server_base.cc:1034] running on GCE node
I20260620 07:17:09.066995 18977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:09.067507 18977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:09.068665 18977 hybrid_clock.cc:648] HybridClock initialized: now 1781939829068633 us; error 42 us; skew 500 ppm
Jun 20 07:17:09 dist-test-slave-j663 krb5kdc[17823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939829, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:09.071262 18977 init.cc:377] Logged in from keytab as oryx/127.11.123.131@KRBTEST.COM (short username oryx)
I20260620 07:17:09.072090 18977 webserver.cc:460] Webserver started at http://127.11.123.131:36145/ using document root <none> and password file <none>
I20260620 07:17:09.072618 18977 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:09.072717 18977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:09.076277 18977 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:09.078238 18992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:09.079366 18977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:09.079501 18977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "1dfdc18ed2bc434abea354b7c13b888a"
format_stamp: "Formatted at 2026-06-20 07:17:07 on dist-test-slave-j663"
server_key: "50d8cf151e5cb0e473af58149e745d1e"
server_key_iv: "eb5dedb8ccbfd1a0ba44618fb17002e0"
server_key_version: "encryptionkey@0"
I20260620 07:17:09.080072 18977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:09.089690 18977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:09.092294 18977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:09.092497 18977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:09.093019 18977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:09.093937 18977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:09.094007 18977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:09.094066 18977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:09.094118 18977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:09.102895 18977 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:36377
I20260620 07:17:09.102919 19105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:36377 every 8 connection(s)
I20260620 07:17:09.103801 18977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:17:09.108588 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 18977
Jun 20 07:17:09 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939829, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
Jun 20 07:17:09 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939829, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.189@KRBTEST.COM
Jun 20 07:17:09 dist-test-slave-j663 krb5kdc[17823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939829, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.188@KRBTEST.COM
I20260620 07:17:09.120973 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.107014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:35445 (local address 127.11.123.188:39431)
0620 07:17:09.107159 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.107163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.107391 (+ 228us) server_negotiation.cc:407] Connection header received
0620 07:17:09.107946 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.107949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.107995 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.108079 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:09.109644 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.110413 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.112053 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.112394 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.112495 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.116377 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:09.116380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:09.116383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:09.116409 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:09.118420 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.119011 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.119015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.119018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.119109 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.119519 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.119522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.119524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.119746 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:09.119833 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.120706 (+ 873us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.120814 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:17:09.122108 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.105903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:33111 (local address 127.11.123.131:56501)
0620 07:17:09.106891 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:09.106904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:09.107451 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:09.107992 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:09.108001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:09.108619 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:09.109461 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.109481 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.110456 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.110459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.111930 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.111940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.112644 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.112649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.112891 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.113972 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:09.113993 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:09.115970 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:09.119311 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:09.119314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:09.119316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:09.119552 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:09.119834 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:09.119837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:09.119840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:09.119951 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:09.120374 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:09.120380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:09.120459 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 07:17:09.120559 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 07:17:09.120734 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":297,"spinlock_wait_cycles":9856,"thread_start_us":155,"threads_started":1}
I20260620 07:17:09.122932 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.105545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:56501 (local address 127.11.123.189:33111)
0620 07:17:09.105685 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.105690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.106974 (+ 1284us) server_negotiation.cc:407] Connection header received
0620 07:17:09.107654 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.107657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.107718 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.107821 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:09.109665 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.110271 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.112067 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.112374 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.112475 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.116141 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:09.116144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:09.116147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:09.116174 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:09.119194 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.119655 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.119658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.119660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.119731 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.120053 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.120056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.120058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.120245 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:09.120333 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.122644 (+ 2311us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.122760 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:17:09.123018 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.105604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:57265 (local address 127.11.123.190:46763)
0620 07:17:09.105791 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.105795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.106561 (+ 766us) server_negotiation.cc:407] Connection header received
0620 07:17:09.107772 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.107774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.107819 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.107907 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:09.110663 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.111453 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.112563 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.112879 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.112954 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.117476 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:09.117480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:09.117482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:09.117507 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:09.119773 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.120292 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.120295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.120296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.120381 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.121032 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.121046 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.121048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.122224 (+ 1176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:09.122451 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.122702 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.122830 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 07:17:09.123605 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.105903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.131:57265)
0620 07:17:09.106391 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:09.106439 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:09.107451 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:09.107994 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:09.108001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:09.108613 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:09.110521 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.110529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.111650 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.111653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.112445 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.112456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.113155 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.113161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.113266 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.113832 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:09.113847 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:09.117309 (+ 3462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:09.119919 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:09.119923 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:09.119925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:09.120181 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:09.120502 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:09.120505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:09.120508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:09.120910 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:09.122354 (+ 1444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:09.122357 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:09.122500 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 07:17:09.123213 (+ 713us) client_negotiation.cc:240] Negotiation successful
0620 07:17:09.123354 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":1154,"spinlock_wait_cycles":26112,"thread_start_us":553,"threads_started":1}
I20260620 07:17:09.123788 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.111556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.11.123.188:39431)
0620 07:17:09.111871 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.111875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.114276 (+ 2401us) server_negotiation.cc:407] Connection header received
0620 07:17:09.114421 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.114425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.114471 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.114564 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:09.116080 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.116833 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.118060 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.118420 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.118501 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.120311 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:09.120314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:09.120317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:09.120341 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:09.121973 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.122432 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.122436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.122437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.122523 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.122834 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.122838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.122840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.123071 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:09.123184 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.123546 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.123653 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20260620 07:17:09.123907 19108 heartbeater.cc:347] Connected to a master server at 127.11.123.189:33111
I20260620 07:17:09.124136 19108 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:09.124213 19107 heartbeater.cc:347] Connected to a master server at 127.11.123.190:46763
I20260620 07:17:09.124306 19107 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:09.124715 19108 heartbeater.cc:511] Master 127.11.123.189:33111 requested a full tablet report, sending...
I20260620 07:17:09.124715 19107 heartbeater.cc:511] Master 127.11.123.190:46763 requested a full tablet report, sending...
I20260620 07:17:09.125638 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.106908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:39431 (local address 127.11.123.131:35445)
0620 07:17:09.107299 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:09.107312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:09.107472 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:09.108100 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:09.108103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:09.108613 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:09.109470 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.109484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.110515 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.110518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.111927 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.111935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.112644 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.112647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.112896 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.113632 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:09.113647 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:09.116236 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:09.118543 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:09.118552 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:09.118572 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:09.118915 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:09.119210 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:09.119214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:09.119219 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:09.119402 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:09.119880 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:09.119888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:09.120211 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 07:17:09.120402 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 07:17:09.125457 (+ 5055us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":5134,"spinlock_wait_cycles":2432,"thread_start_us":85,"threads_started":1}
I20260620 07:17:09.125797 17868 ts_manager.cc:194] Re-registered known tserver with Master: 1dfdc18ed2bc434abea354b7c13b888a (127.11.123.131:36377)
I20260620 07:17:09.125900 18461 ts_manager.cc:194] Registered new tserver with Master: 1dfdc18ed2bc434abea354b7c13b888a (127.11.123.131:36377)
I20260620 07:17:09.126525 17868 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:57265
I20260620 07:17:09.127122 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.111660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.11.123.190:46763)
0620 07:17:09.112254 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.112266 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.112840 (+ 574us) server_negotiation.cc:407] Connection header received
0620 07:17:09.112880 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.112883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.112929 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.113006 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:09.114459 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.115225 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.117284 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.117693 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.117789 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.122033 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:09.122038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:09.122041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:09.122067 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:09.124964 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.125429 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.125435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.125440 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.125533 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.125828 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.125831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.125833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.126061 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:09.126198 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.126842 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.126964 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":84,"threads_started":1}
I20260620 07:17:09.127370 19106 heartbeater.cc:347] Connected to a master server at 127.11.123.188:39431
I20260620 07:17:09.127439 19106 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:09.127939 19106 heartbeater.cc:511] Master 127.11.123.188:39431 requested a full tablet report, sending...
I20260620 07:17:09.128556 18645 ts_manager.cc:194] Registered new tserver with Master: 1dfdc18ed2bc434abea354b7c13b888a (127.11.123.131:36377)
I20260620 07:17:09.130226 17868 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 07:17:09.133262 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.110622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.11.123.189:33111)
0620 07:17:09.114674 (+ 4052us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.114678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.114690 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:17:09.114724 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.114727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.114765 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.114844 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:09.126408 (+ 11564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.127123 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.128206 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.128548 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.128641 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.130196 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:09.130199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:09.130200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:09.130220 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:09.131645 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.132046 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.132049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.132051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.132140 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.132412 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.132415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.132418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.132611 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:09.132745 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.132958 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.133096 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3983,"thread_start_us":84,"threads_started":1}
W20260620 07:17:09.133394 17868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:17:09.136291 18427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: NON_VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: true } } }
I20260620 07:17:09.136409 18427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:09.139847 18428 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280: attempting to promote NON_VOTER d36ebdb692ec4976980191ef158e3e29 to VOTER
I20260620 07:17:09.140139 18428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } }
I20260620 07:17:09.142355 18479 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 FOLLOWER]: Refusing update from remote peer 20933201ca394c1e97bc08ecdc8d4280: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 07:17:09.142704 18427 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 07:17:09.144117 18655 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 LEARNER]: Refusing update from remote peer 20933201ca394c1e97bc08ecdc8d4280: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 07:17:09.144462 18427 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 07:17:09.145099 19123 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d36ebdb692ec4976980191ef158e3e29 (127.11.123.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } } }
I20260620 07:17:09.145470 18479 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d36ebdb692ec4976980191ef158e3e29 (127.11.123.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } } }
I20260620 07:17:09.146533 18428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } } }
I20260620 07:17:09.146646 18428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:09.146548 19123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } } }
I20260620 07:17:09.147118 19123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:09.147421 18428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } } }
I20260620 07:17:09.147521 18428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:09.150686 18655 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d36ebdb692ec4976980191ef158e3e29 (127.11.123.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } } }
I20260620 07:17:09.151211 19132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } } }
I20260620 07:17:09.151474 19132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:17:09.152447 18695 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20933201ca394c1e97bc08ecdc8d4280" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } } }
I20260620 07:17:09.152566 18695 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:17:09.154462 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.149217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:36377 (local address 127.0.0.1:46962)
0620 07:17:09.149400 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:09.149418 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:09.149499 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:09.150190 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:09.150193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:09.150212 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:09.150328 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.150334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.151874 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.151878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.152993 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.153003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.154050 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.154054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.154134 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.154154 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:17:09.154215 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 07:17:09.154341 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":44}
I20260620 07:17:09.154477 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.149353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.11.123.131:36377)
0620 07:17:09.149659 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.149675 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.149700 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:17:09.149780 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.149785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.150015 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.150478 (+ 463us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:09.150620 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.151742 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.153395 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.153927 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.154073 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.154209 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.154258 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.154345 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
I20260620 07:17:09.156733 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.149751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:42833 (local address 127.0.0.1:33706)
0620 07:17:09.151682 (+ 1931us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:09.151693 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:09.151804 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:09.152290 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:09.152293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:09.152309 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:09.152404 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.152410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.154169 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.154172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.155567 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.155575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.156357 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.156363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.156464 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.156489 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:17:09.156535 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 07:17:09.156604 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1878,"thread_start_us":99,"threads_started":1}
I20260620 07:17:09.157407 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.149907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.11.123.130:42833)
0620 07:17:09.150337 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.150351 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.151789 (+ 1438us) server_negotiation.cc:407] Connection header received
0620 07:17:09.151916 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.151922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.152141 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.152536 (+ 395us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:09.152979 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.154049 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.155681 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.156192 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.157069 (+ 877us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.157203 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.157241 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.157303 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":110,"threads_started":1}
I20260620 07:17:09.157382 19040 tablet_service.cc:1467] Processing CreateTablet for tablet 73d060f98bf54c2c8be48ad1f5b47104 (DEFAULT_TABLE table=test-table [id=2e9018286094493898578941024ca08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:09.158094 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.149791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:41141 (local address 127.0.0.1:33104)
0620 07:17:09.151983 (+ 2192us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:09.151995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:09.152064 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:09.153751 (+ 1687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:09.153754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:09.153771 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:09.153857 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.153862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.155419 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.155423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.156492 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.156503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.157724 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.157728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.157808 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.157829 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:17:09.157904 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 07:17:09.157969 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2141,"thread_start_us":77,"threads_started":1}
I20260620 07:17:09.158488 19040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d060f98bf54c2c8be48ad1f5b47104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:09.158569 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.150160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.11.123.129:41141)
0620 07:17:09.152322 (+ 2162us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.152334 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.152354 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:17:09.152422 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.152428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.152638 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.152775 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:09.154008 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.155305 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.156635 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.157444 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.157562 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.157664 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.158336 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.158429 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2039,"thread_start_us":106,"threads_started":1}
I20260620 07:17:09.159955 18903 tablet_service.cc:1467] Processing CreateTablet for tablet 73d060f98bf54c2c8be48ad1f5b47104 (DEFAULT_TABLE table=test-table [id=2e9018286094493898578941024ca08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:09.160928 18903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d060f98bf54c2c8be48ad1f5b47104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:09.161197 18766 tablet_service.cc:1467] Processing CreateTablet for tablet 73d060f98bf54c2c8be48ad1f5b47104 (DEFAULT_TABLE table=test-table [id=2e9018286094493898578941024ca08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:09.162091 18766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d060f98bf54c2c8be48ad1f5b47104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:09.165285 19138 tablet_bootstrap.cc:492] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a: Bootstrap starting.
I20260620 07:17:09.167292 19138 tablet_bootstrap.cc:654] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:09.168043 19138 log.cc:824] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:09.169651 19140 tablet_bootstrap.cc:492] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2: Bootstrap starting.
I20260620 07:17:09.169872 19138 tablet_bootstrap.cc:492] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a: No bootstrap required, opened a new log
I20260620 07:17:09.170064 19138 ts_tablet_manager.cc:1397] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:17:09.171360 19141 tablet_bootstrap.cc:492] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Bootstrap starting.
I20260620 07:17:09.171993 19140 tablet_bootstrap.cc:654] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:09.172793 19140 log.cc:824] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:09.173221 19141 tablet_bootstrap.cc:654] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:09.173926 19141 log.cc:824] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:09.174463 19138 raft_consensus.cc:348] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:09.174722 19138 raft_consensus.cc:374] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:09.174789 19138 raft_consensus.cc:729] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dfdc18ed2bc434abea354b7c13b888a, State: Initialized, Role: FOLLOWER
I20260620 07:17:09.174827 19140 tablet_bootstrap.cc:492] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2: No bootstrap required, opened a new log
I20260620 07:17:09.174997 19140 ts_tablet_manager.cc:1397] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 07:17:09.175277 19138 consensus_queue.cc:260] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:09.176252 19138 ts_tablet_manager.cc:1428] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 07:17:09.176326 19141 tablet_bootstrap.cc:492] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: No bootstrap required, opened a new log
I20260620 07:17:09.176499 19141 ts_tablet_manager.cc:1397] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 07:17:09.176556 19107 heartbeater.cc:503] Master 127.11.123.190:46763 was elected leader, sending a full tablet report...
I20260620 07:17:09.179319 19140 raft_consensus.cc:348] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:09.179600 19140 raft_consensus.cc:374] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:09.179647 19140 raft_consensus.cc:729] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb35bc6d8714b7599cba2fce4b653c2, State: Initialized, Role: FOLLOWER
I20260620 07:17:09.180081 19140 consensus_queue.cc:260] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:09.181048 19140 ts_tablet_manager.cc:1428] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 07:17:09.181407 18833 heartbeater.cc:503] Master 127.11.123.190:46763 was elected leader, sending a full tablet report...
I20260620 07:17:09.181784 19141 raft_consensus.cc:348] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:09.182026 19141 raft_consensus.cc:374] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:09.182080 19141 raft_consensus.cc:729] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a5acc88ff0c4c86a67e6b97ea1cee0c, State: Initialized, Role: FOLLOWER
I20260620 07:17:09.182513 19141 consensus_queue.cc:260] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:09.183849 19141 ts_tablet_manager.cc:1428] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260620 07:17:09.184983 18970 heartbeater.cc:503] Master 127.11.123.190:46763 was elected leader, sending a full tablet report...
W20260620 07:17:09.221899 18835 tablet.cc:2369] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:17:09.342332 19144 raft_consensus.cc:482] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:17:09.342514 19144 raft_consensus.cc:504] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:09.343564 19144 leader_election.cc:288] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbb35bc6d8714b7599cba2fce4b653c2 (127.11.123.129:41141), 1a5acc88ff0c4c86a67e6b97ea1cee0c (127.11.123.130:42833)
I20260620 07:17:09.348618 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.343921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:56057 (local address 127.11.123.129:41141)
0620 07:17:09.344043 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.344048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.344149 (+ 101us) server_negotiation.cc:407] Connection header received
0620 07:17:09.344349 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.344355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.344401 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.344476 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:09.344752 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.345568 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.347342 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.347956 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.348070 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.348128 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.348424 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.348493 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:17:09.348618 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.343932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:45627 (local address 127.11.123.130:42833)
0620 07:17:09.344079 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.344083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.344128 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:17:09.344216 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.344219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.344259 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.344365 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:09.344753 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.345568 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.347342 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.347948 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.348071 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.348123 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.348424 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.348493 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:17:09.348623 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.343790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:41141 (local address 127.11.123.131:56057)
0620 07:17:09.344061 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:09.344074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:09.344190 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:09.344495 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:09.344497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:09.344509 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:09.344629 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.344636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.345694 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.345698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.347231 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.347237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.348071 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.348075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.348275 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.348320 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 07:17:09.348394 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 07:17:09.348486 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":258,"spinlock_wait_cycles":768}
I20260620 07:17:09.349023 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.343790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:42833 (local address 127.11.123.131:45627)
0620 07:17:09.344002 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:09.344019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:09.344118 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:09.344438 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:09.344441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:09.344484 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:09.344630 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.344636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.345694 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.345698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.347212 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:09.347222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.348071 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.348075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:09.348270 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.348315 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 07:17:09.348394 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 07:17:09.348492 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":258}
I20260620 07:17:09.349290 18786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d060f98bf54c2c8be48ad1f5b47104" candidate_uuid: "1dfdc18ed2bc434abea354b7c13b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" is_pre_election: true
I20260620 07:17:09.349572 18786 raft_consensus.cc:2454] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dfdc18ed2bc434abea354b7c13b888a in term 0.
I20260620 07:17:09.349651 18923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d060f98bf54c2c8be48ad1f5b47104" candidate_uuid: "1dfdc18ed2bc434abea354b7c13b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" is_pre_election: true
I20260620 07:17:09.349962 18923 raft_consensus.cc:2454] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dfdc18ed2bc434abea354b7c13b888a in term 0.
I20260620 07:17:09.350045 18996 leader_election.cc:302] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dfdc18ed2bc434abea354b7c13b888a, cbb35bc6d8714b7599cba2fce4b653c2; no voters:
I20260620 07:17:09.350313 19144 raft_consensus.cc:2781] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:17:09.350407 19144 raft_consensus.cc:482] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:17:09.350451 19144 raft_consensus.cc:3037] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:09.351438 19144 raft_consensus.cc:504] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:09.351740 19144 leader_election.cc:288] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [CANDIDATE]: Term 1 election: Requested vote from peers cbb35bc6d8714b7599cba2fce4b653c2 (127.11.123.129:41141), 1a5acc88ff0c4c86a67e6b97ea1cee0c (127.11.123.130:42833)
I20260620 07:17:09.352065 18923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d060f98bf54c2c8be48ad1f5b47104" candidate_uuid: "1dfdc18ed2bc434abea354b7c13b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c"
I20260620 07:17:09.352058 18786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d060f98bf54c2c8be48ad1f5b47104" candidate_uuid: "1dfdc18ed2bc434abea354b7c13b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb35bc6d8714b7599cba2fce4b653c2"
I20260620 07:17:09.352175 18923 raft_consensus.cc:3037] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:09.352187 18786 raft_consensus.cc:3037] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:09.353382 18923 raft_consensus.cc:2454] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dfdc18ed2bc434abea354b7c13b888a in term 1.
I20260620 07:17:09.353382 18786 raft_consensus.cc:2454] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dfdc18ed2bc434abea354b7c13b888a in term 1.
I20260620 07:17:09.353634 18996 leader_election.cc:302] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dfdc18ed2bc434abea354b7c13b888a, cbb35bc6d8714b7599cba2fce4b653c2; no voters:
I20260620 07:17:09.353814 19144 raft_consensus.cc:2781] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:09.354038 19144 raft_consensus.cc:686] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 1 LEADER]: Becoming Leader. State: Replica: 1dfdc18ed2bc434abea354b7c13b888a, State: Running, Role: LEADER
I20260620 07:17:09.354372 19144 consensus_queue.cc:237] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
W20260620 07:17:09.355048 19109 tablet.cc:2369] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:17:09.357785 17868 catalog_manager.cc:5495] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dfdc18ed2bc434abea354b7c13b888a (127.11.123.131). New cstate: current_term: 1 leader_uuid: "1dfdc18ed2bc434abea354b7c13b888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } health_report { overall_health: UNKNOWN } } }
I20260620 07:17:09.392570 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.388928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.11.123.131:36377)
0620 07:17:09.389109 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.389116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.389150 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:17:09.389259 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.389262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.389330 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.389450 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:17:09.389689 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.390210 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.391036 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.391247 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.391322 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.391567 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:17:09.392095 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:17:09.392208 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.392326 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.392377 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:17:09.398227 18923 raft_consensus.cc:1261] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 1 FOLLOWER]: Refusing update from remote peer 1dfdc18ed2bc434abea354b7c13b888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:09.398227 18786 raft_consensus.cc:1261] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2 [term 1 FOLLOWER]: Refusing update from remote peer 1dfdc18ed2bc434abea354b7c13b888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:09.398914 19144 consensus_queue.cc:1035] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:09.399104 19147 consensus_queue.cc:1035] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:09.406555 19150 mvcc.cc:205] Tried to move back new op lower bound from 7298825541208227840 to 7298825541038190592. Current Snapshot: MvccSnapshot[applied={T|T < 7298825541208227840}]
I20260620 07:17:09.407986 19152 mvcc.cc:205] Tried to move back new op lower bound from 7298825541208227840 to 7298825541038190592. Current Snapshot: MvccSnapshot[applied={T|T < 7298825541208227840}]
I20260620 07:17:09.417047 17867 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:09.417271 17867 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:09.422276 17867 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20933201ca394c1e97bc08ecdc8d4280: Sending DeleteTablet for 3 replicas of tablet 73d060f98bf54c2c8be48ad1f5b47104
I20260620 07:17:09.423121 19040 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d060f98bf54c2c8be48ad1f5b47104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:09 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:46962
I20260620 07:17:09.423237 18766 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d060f98bf54c2c8be48ad1f5b47104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:09 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:33104
I20260620 07:17:09.423555 19039 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d060f98bf54c2c8be48ad1f5b47104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:09 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:46962
I20260620 07:17:09.423612 19160 tablet_replica.cc:330] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a: stopping tablet replica
I20260620 07:17:09.424001 19160 raft_consensus.cc:2229] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:17:09.424021 18903 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d060f98bf54c2c8be48ad1f5b47104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:09 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:33706
I20260620 07:17:09.424453 19160 raft_consensus.cc:2258] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:09.424681 19161 tablet_replica.cc:330] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2: stopping tablet replica
I20260620 07:17:09.424953 19161 raft_consensus.cc:2229] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:09.425307 19161 raft_consensus.cc:2258] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:09.425946 19162 tablet_replica.cc:330] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: stopping tablet replica
I20260620 07:17:09.426237 19162 raft_consensus.cc:2229] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:09.426651 19162 raft_consensus.cc:2258] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:09.427132 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 17836
I20260620 07:17:09.427635 19160 ts_tablet_manager.cc:1905] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:09.430533 19160 ts_tablet_manager.cc:1918] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:09.431046 19161 ts_tablet_manager.cc:1905] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:09.433703 19161 ts_tablet_manager.cc:1918] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:09.433876 19160 log.cc:1196] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/73d060f98bf54c2c8be48ad1f5b47104
I20260620 07:17:09.434255 19160 ts_tablet_manager.cc:1939] T 73d060f98bf54c2c8be48ad1f5b47104 P 1dfdc18ed2bc434abea354b7c13b888a: Deleting consensus metadata
I20260620 07:17:09.434386 19161 log.cc:1196] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-0/wal/wals/73d060f98bf54c2c8be48ad1f5b47104
W20260620 07:17:09.435689 18995 connection.cc:535] server connection from 127.0.0.1:46962 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 07:17:09.436108 19161 ts_tablet_manager.cc:1939] T 73d060f98bf54c2c8be48ad1f5b47104 P cbb35bc6d8714b7599cba2fce4b653c2: Deleting consensus metadata
W20260620 07:17:09.437212 18719 connection.cc:441] server connection from 127.0.0.1:33104 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33104 (request call id 1) could send its response
W20260620 07:17:09.437402 18995 connection.cc:441] server connection from 127.0.0.1:46962 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46962 (request call id 2) could send its response
I20260620 07:17:09.437968 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.426115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.11.123.189:33111)
0620 07:17:09.426223 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.426227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.426241 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:17:09.426277 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.426280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.426324 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.426404 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:09.428797 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.429734 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.431095 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.431558 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.431652 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.433778 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:09.433784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:09.433791 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:09.433823 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:09.435643 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.436195 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.436199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.436201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.436302 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.436654 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.436658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.436660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.436891 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:09.436999 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.437676 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.437809 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 07:17:09.442698 19162 ts_tablet_manager.cc:1905] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:09.445367 19162 ts_tablet_manager.cc:1918] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:09.445465 19162 log.cc:1196] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/73d060f98bf54c2c8be48ad1f5b47104
I20260620 07:17:09.445820 19162 ts_tablet_manager.cc:1939] T 73d060f98bf54c2c8be48ad1f5b47104 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Deleting consensus metadata
W20260620 07:17:09.446915 18856 connection.cc:441] server connection from 127.0.0.1:33706 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33706 (request call id 1) could send its response
I20260620 07:17:09.447727 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.426224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.11.123.188:39431)
0620 07:17:09.426339 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.426343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.438278 (+ 11935us) server_negotiation.cc:407] Connection header received
0620 07:17:09.438464 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.438468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.438529 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.438581 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:09.439444 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.440274 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.441453 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.441811 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.441889 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.443642 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:09.443646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:09.443650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:09.443674 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:09.445369 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.446239 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.446243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.446244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.446326 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.446609 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.446613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.446615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.446833 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:09.446933 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.447399 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.447528 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 07:17:09.453405 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.440549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.11.123.188:39431)
0620 07:17:09.440656 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.440661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.441370 (+ 709us) server_negotiation.cc:407] Connection header received
0620 07:17:09.441548 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.441550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.441597 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.441655 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:09.442976 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.443693 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.444792 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.445140 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.445229 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.449217 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:09.449221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:09.449223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:09.449247 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:09.450927 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.451405 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.451408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.451410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.451493 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.452395 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.452400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.452403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.452626 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:09.452768 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.452998 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.453263 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 07:17:09.455046 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:09.440861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.11.123.189:33111)
0620 07:17:09.440968 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:09.440972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:09.445918 (+ 4946us) server_negotiation.cc:407] Connection header received
0620 07:17:09.446094 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:09.446097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:09.446138 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:09.446186 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:09.447978 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.448645 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.449795 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:09.450148 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:09.450244 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:09.451634 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:09.451638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:09.451641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:09.451666 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:09.453299 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.453967 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.453970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.453972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.454052 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:09.454329 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:09.454332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:09.454333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:09.454504 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:09.454603 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:09.454824 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 07:17:09.454907 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 07:17:10.410820 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.410380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.130:53163)
0620 07:17:10.410641 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.410724 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
W20260620 07:17:10.411154 18970 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:46763 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
I20260620 07:17:10.418574 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.418092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.129:38469)
0620 07:17:10.418348 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.418455 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
W20260620 07:17:10.419168 18833 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:46763 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
I20260620 07:17:10.423947 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.423520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.131:36773)
0620 07:17:10.423748 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.423826 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
W20260620 07:17:10.424243 19107 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:46763 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
I20260620 07:17:10.921849 19174 raft_consensus.cc:482] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 20933201ca394c1e97bc08ecdc8d4280)
I20260620 07:17:10.922076 19174 raft_consensus.cc:504] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } }
I20260620 07:17:10.923004 19175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 20933201ca394c1e97bc08ecdc8d4280)
I20260620 07:17:10.923249 19174 leader_election.cc:288] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20933201ca394c1e97bc08ecdc8d4280 (127.11.123.190:46763), 2045f665fa4b4c2499e7ff0cbca5b412 (127.11.123.189:33111)
I20260620 07:17:10.923187 19175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } }
I20260620 07:17:10.923784 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.923373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46798)
0620 07:17:10.923587 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.923687 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
W20260620 07:17:10.923974 18640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
I20260620 07:17:10.924561 19175 leader_election.cc:288] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20933201ca394c1e97bc08ecdc8d4280 (127.11.123.190:46763), d36ebdb692ec4976980191ef158e3e29 (127.11.123.188:39431)
I20260620 07:17:10.925256 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.924973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46826)
0620 07:17:10.925129 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.925188 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 07:17:10.925391 18640 leader_election.cc:334] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 20933201ca394c1e97bc08ecdc8d4280 (127.11.123.190:46763): Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
I20260620 07:17:10.925518 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.924699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46812)
0620 07:17:10.925340 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.925406 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":57,"threads_started":1}
W20260620 07:17:10.925664 18456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
I20260620 07:17:10.926390 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.926192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46834)
0620 07:17:10.926274 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.926327 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 07:17:10.926513 18456 leader_election.cc:334] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 20933201ca394c1e97bc08ecdc8d4280 (127.11.123.190:46763): Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
I20260620 07:17:10.928126 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.923382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:33111 (local address 127.0.0.1:60886)
0620 07:17:10.923944 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.923958 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:10.924075 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:10.924584 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:10.924590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:10.924614 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:10.924720 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:10.924727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.926011 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.926017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:10.927019 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:10.927029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.927712 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.927716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:10.927804 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:10.927834 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:17:10.927938 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 07:17:10.928008 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":3,"thread_start_us":57,"threads_started":1}
I20260620 07:17:10.928125 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.923677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.11.123.189:33111)
0620 07:17:10.924314 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:10.924318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:10.924335 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:17:10.924384 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:10.924387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:10.924443 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:10.924537 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:10.924853 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.925901 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.927141 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.927590 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.927722 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:10.927771 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:10.927948 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 07:17:10.928015 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":161,"thread_start_us":58,"threads_started":1}
I20260620 07:17:10.928517 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d36ebdb692ec4976980191ef158e3e29" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" is_pre_election: true
I20260620 07:17:10.928709 18478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d36ebdb692ec4976980191ef158e3e29 in term 1.
I20260620 07:17:10.929049 18639 leader_election.cc:302] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2045f665fa4b4c2499e7ff0cbca5b412, d36ebdb692ec4976980191ef158e3e29; no voters: 20933201ca394c1e97bc08ecdc8d4280
I20260620 07:17:10.929291 19174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 07:17:10.929350 19174 raft_consensus.cc:482] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 20933201ca394c1e97bc08ecdc8d4280)
I20260620 07:17:10.929389 19174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:17:10.929575 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.925000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:39431 (local address 127.0.0.1:35708)
0620 07:17:10.926055 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.926069 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:10.926354 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:10.926714 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:10.926717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:10.926733 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:10.926826 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:10.926831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.927553 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.927558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:10.928458 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:10.928468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.929208 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.929212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:10.929300 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:10.929320 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:17:10.929394 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 07:17:10.929462 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"thread_start_us":53,"threads_started":1}
I20260620 07:17:10.929886 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.925128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.11.123.188:39431)
0620 07:17:10.926167 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:10.926176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:10.926319 (+ 143us) server_negotiation.cc:407] Connection header received
0620 07:17:10.926429 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:10.926432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:10.926619 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:10.926707 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:10.926946 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.927434 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.928586 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.929073 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.929217 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:10.929259 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:10.929729 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 07:17:10.929800 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":114,"threads_started":1}
I20260620 07:17:10.930249 18655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d36ebdb692ec4976980191ef158e3e29" is_pre_election: true
I20260620 07:17:10.930768 19174 raft_consensus.cc:504] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } }
I20260620 07:17:10.931006 18655 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2045f665fa4b4c2499e7ff0cbca5b412 in current term 2: Already voted for candidate d36ebdb692ec4976980191ef158e3e29 in this term.
I20260620 07:17:10.931136 19174 leader_election.cc:288] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [CANDIDATE]: Term 2 election: Requested vote from peers 20933201ca394c1e97bc08ecdc8d4280 (127.11.123.190:46763), 2045f665fa4b4c2499e7ff0cbca5b412 (127.11.123.189:33111)
I20260620 07:17:10.931445 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d36ebdb692ec4976980191ef158e3e29" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2045f665fa4b4c2499e7ff0cbca5b412"
I20260620 07:17:10.931489 18456 leader_election.cc:302] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2045f665fa4b4c2499e7ff0cbca5b412; no voters: 20933201ca394c1e97bc08ecdc8d4280, d36ebdb692ec4976980191ef158e3e29
I20260620 07:17:10.931566 18478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:17:10.931972 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.931341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46848)
0620 07:17:10.931877 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.931919 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481}
I20260620 07:17:10.932507 18478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d36ebdb692ec4976980191ef158e3e29 in term 2.
I20260620 07:17:10.932545 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.932347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46864)
0620 07:17:10.932449 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.932492 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 07:17:10.932647 18640 leader_election.cc:334] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 20933201ca394c1e97bc08ecdc8d4280 (127.11.123.190:46763): Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
I20260620 07:17:10.932775 19175 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:17:10.932842 18639 leader_election.cc:302] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2045f665fa4b4c2499e7ff0cbca5b412, d36ebdb692ec4976980191ef158e3e29; no voters: 20933201ca394c1e97bc08ecdc8d4280
I20260620 07:17:10.933025 19174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:17:10.933264 19174 raft_consensus.cc:686] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [term 2 LEADER]: Becoming Leader. State: Replica: d36ebdb692ec4976980191ef158e3e29, State: Running, Role: LEADER
I20260620 07:17:10.933465 19174 consensus_queue.cc:237] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } }
I20260620 07:17:10.934527 19184 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d36ebdb692ec4976980191ef158e3e29. Latest consensus state: current_term: 2 leader_uuid: "d36ebdb692ec4976980191ef158e3e29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } } }
I20260620 07:17:10.934661 19184 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:10.934926 19186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:10.937131 19186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:10.937503 19186 catalog_manager.cc:1259] Loaded cluster ID: 8a09ee15963f4a8da59c69cf1b321621
I20260620 07:17:10.937587 19186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:10.938102 19186 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:10.938531 19186 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: Loaded TSK: 0
I20260620 07:17:10.939186 19186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:17:10.971536 18647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:17:10.974660 18647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:17:10.977473 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.977199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46886)
0620 07:17:10.977334 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.977395 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 07:17:10.977500 18478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [term 2 FOLLOWER]: Refusing update from remote peer d36ebdb692ec4976980191ef158e3e29: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 07:17:10.977985 19184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 07:17:10.978266 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.978040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.0.0.1:46902)
0620 07:17:10.978166 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.978215 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 07:17:10.978380 18640 consensus_peers.cc:487] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 -> Peer 20933201ca394c1e97bc08ecdc8d4280 (127.11.123.190:46763): Couldn't send request to peer 20933201ca394c1e97bc08ecdc8d4280. Status: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 07:17:10.979420 19175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d36ebdb692ec4976980191ef158e3e29. Latest consensus state: current_term: 2 leader_uuid: "d36ebdb692ec4976980191ef158e3e29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } } }
I20260620 07:17:10.979532 19175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:17:10.980576 19184 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d36ebdb692ec4976980191ef158e3e29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } } }
I20260620 07:17:10.980723 19184 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:10.981175 19175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d36ebdb692ec4976980191ef158e3e29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20933201ca394c1e97bc08ecdc8d4280" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46763 } } peers { permanent_uuid: "2045f665fa4b4c2499e7ff0cbca5b412" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 33111 } attrs { promote: false } } peers { permanent_uuid: "d36ebdb692ec4976980191ef158e3e29" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 39431 } attrs { promote: false } } }
I20260620 07:17:10.981312 19175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2045f665fa4b4c2499e7ff0cbca5b412 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:17:10.981431 19188 mvcc.cc:205] Tried to move back new op lower bound from 7298825547679117312 to 7298825547505418240. Current Snapshot: MvccSnapshot[applied={T|T < 7298825547679117312}]
I20260620 07:17:10.990753 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.986387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:36377 (local address 127.0.0.1:46984)
0620 07:17:10.986709 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.986723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:10.986800 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:10.987324 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:10.987327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:10.987344 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:10.987473 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:10.987479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.988757 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.988761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:10.989650 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:10.989658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.990303 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.990307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:10.990438 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:10.990469 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:17:10.990560 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:17:10.990628 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":26}
I20260620 07:17:10.991478 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.986479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.11.123.131:36377)
0620 07:17:10.986750 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:10.986801 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:10.986819 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:17:10.987155 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:10.987159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:10.987215 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:10.987305 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:10.987583 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.988602 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.989770 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.990186 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.990321 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:10.990380 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:10.991258 (+ 878us) server_negotiation.cc:299] Negotiation successful
0620 07:17:10.991343 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
I20260620 07:17:10.991667 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.986681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:41141 (local address 127.0.0.1:33110)
0620 07:17:10.986932 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.986942 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:10.987026 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:10.987802 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:10.987806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:10.987823 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:10.987922 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:10.987928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.989127 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.989130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:10.990391 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:10.990397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.991266 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.991269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:10.991384 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:10.991408 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:17:10.991456 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 07:17:10.991531 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":76}
I20260620 07:17:10.991796 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.987266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.11.123.130:42833)
0620 07:17:10.987552 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:10.987560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:10.987572 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:17:10.987618 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:10.987620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:10.987668 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:10.987764 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:10.988185 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.988886 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.990526 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.991169 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.991287 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:10.991383 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:10.991570 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 07:17:10.991652 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":64,"threads_started":1}
I20260620 07:17:10.991998 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.987013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:42833 (local address 127.0.0.1:33716)
0620 07:17:10.987224 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:10.987234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:10.987323 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:10.987781 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:10.987784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:10.987803 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:10.988040 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:10.988049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.989025 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.989028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:10.990381 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:10.990391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.991274 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.991278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:10.991429 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:10.991450 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:17:10.991544 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 07:17:10.991619 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":235,"spinlock_wait_cycles":3456,"thread_start_us":83,"threads_started":1}
I20260620 07:17:10.992151 19039 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc42ae465d849d38dcc4947a5519349 (DEFAULT_TABLE table=test-table [id=dc215a4b78e1479ca16948812b839298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:10.992280 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.986927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.11.123.129:41141)
0620 07:17:10.987512 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:10.987516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:10.987530 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:17:10.987583 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:10.987587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:10.987683 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:10.988060 (+ 377us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:10.988178 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.988998 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.990483 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:10.991131 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:10.992004 (+ 873us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:10.992050 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:10.992083 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 07:17:10.992152 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":87,"threads_started":1}
I20260620 07:17:10.992470 19039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc42ae465d849d38dcc4947a5519349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:10.992535 18903 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc42ae465d849d38dcc4947a5519349 (DEFAULT_TABLE table=test-table [id=dc215a4b78e1479ca16948812b839298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:10.992710 18766 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc42ae465d849d38dcc4947a5519349 (DEFAULT_TABLE table=test-table [id=dc215a4b78e1479ca16948812b839298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:10.992794 18903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc42ae465d849d38dcc4947a5519349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:10.992937 18766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc42ae465d849d38dcc4947a5519349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:10.994966 19197 tablet_bootstrap.cc:492] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a: Bootstrap starting.
I20260620 07:17:10.995829 19198 tablet_bootstrap.cc:492] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Bootstrap starting.
I20260620 07:17:10.995882 19197 tablet_bootstrap.cc:654] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:10.996727 19198 tablet_bootstrap.cc:654] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:10.997069 19197 tablet_bootstrap.cc:492] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a: No bootstrap required, opened a new log
I20260620 07:17:10.997195 19197 ts_tablet_manager.cc:1397] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:10.997287 19199 tablet_bootstrap.cc:492] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2: Bootstrap starting.
I20260620 07:17:10.997700 19197 raft_consensus.cc:348] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:10.997829 19197 raft_consensus.cc:374] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:10.997869 19197 raft_consensus.cc:729] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dfdc18ed2bc434abea354b7c13b888a, State: Initialized, Role: FOLLOWER
I20260620 07:17:10.997987 19197 consensus_queue.cc:260] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:10.998086 19199 tablet_bootstrap.cc:654] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:10.998342 19197 ts_tablet_manager.cc:1428] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:17:10.999451 19198 tablet_bootstrap.cc:492] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: No bootstrap required, opened a new log
I20260620 07:17:10.999567 19198 ts_tablet_manager.cc:1397] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 07:17:11.000072 19198 raft_consensus.cc:348] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:11.000195 19198 raft_consensus.cc:374] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:11.000254 19198 raft_consensus.cc:729] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a5acc88ff0c4c86a67e6b97ea1cee0c, State: Initialized, Role: FOLLOWER
I20260620 07:17:11.000389 19198 consensus_queue.cc:260] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:11.000733 19198 ts_tablet_manager.cc:1428] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:17:11.000875 19106 heartbeater.cc:503] Master 127.11.123.188:39431 was elected leader, sending a full tablet report...
I20260620 07:17:11.001881 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.001159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.130:45281)
0620 07:17:11.001417 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.001477 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":115,"threads_started":1}
I20260620 07:17:11.002661 18969 heartbeater.cc:503] Master 127.11.123.188:39431 was elected leader, sending a full tablet report...
I20260620 07:17:11.003123 19199 tablet_bootstrap.cc:492] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2: No bootstrap required, opened a new log
I20260620 07:17:11.003186 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.002333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.130:46941)
0620 07:17:11.003055 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.003108 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":82,"threads_started":1}
I20260620 07:17:11.003217 19199 ts_tablet_manager.cc:1397] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 07:17:11.003646 19199 raft_consensus.cc:348] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:11.003798 19199 raft_consensus.cc:374] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:11.003841 19199 raft_consensus.cc:729] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb35bc6d8714b7599cba2fce4b653c2, State: Initialized, Role: FOLLOWER
I20260620 07:17:11.003976 19199 consensus_queue.cc:260] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:11.004276 19199 ts_tablet_manager.cc:1428] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:17:11.004812 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:10.998760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.131:51479)
0620 07:17:11.004542 (+ 5782us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.004683 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5701,"thread_start_us":120,"threads_started":1}
I20260620 07:17:11.005295 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.004608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.129:33119)
0620 07:17:11.005132 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.005193 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":100,"threads_started":1}
I20260620 07:17:11.005626 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.005241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.131:40025)
0620 07:17:11.005490 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.005541 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260620 07:17:11.006065 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.005690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.129:35307)
0620 07:17:11.005921 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.005976 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
I20260620 07:17:11.006099 18832 heartbeater.cc:503] Master 127.11.123.188:39431 was elected leader, sending a full tablet report...
I20260620 07:17:11.014711 19208 raft_consensus.cc:482] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:17:11.014838 19208 raft_consensus.cc:504] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:11.015878 19208 leader_election.cc:288] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dfdc18ed2bc434abea354b7c13b888a (127.11.123.131:36377), 1a5acc88ff0c4c86a67e6b97ea1cee0c (127.11.123.130:42833)
I20260620 07:17:11.021169 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.016179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:51221 (local address 127.11.123.131:36377)
0620 07:17:11.016301 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:11.016306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:11.016355 (+ 49us) server_negotiation.cc:407] Connection header received
0620 07:17:11.016467 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:11.016471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:11.016520 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:11.016606 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:11.017065 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:11.017869 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:11.019758 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:11.020391 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:11.020517 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:11.020569 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:11.020940 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 07:17:11.021012 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 07:17:11.021167 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.016335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:50259 (local address 127.11.123.130:42833)
0620 07:17:11.016439 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:11.016442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:11.016453 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:17:11.016492 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:11.016494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:11.016533 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:11.016613 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:11.017065 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:11.017869 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:11.019758 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:11.020391 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:11.020518 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:11.020569 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:11.020940 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 07:17:11.021012 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:17:11.021236 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.016077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:36377 (local address 127.11.123.129:51221)
0620 07:17:11.016294 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.016306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:11.016381 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:11.016632 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:11.016635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:11.016666 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:11.016952 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:11.016958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:11.017992 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:11.017996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:11.019610 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:11.019621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:11.020530 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:11.020535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:11.020758 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:11.020829 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 07:17:11.020910 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 07:17:11.021066 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":433,"spinlock_wait_cycles":6784}
I20260620 07:17:11.021646 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.016078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:42833 (local address 127.11.123.129:50259)
0620 07:17:11.016227 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.016243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:11.016342 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:11.016632 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:11.016635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:11.016665 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:11.016951 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:11.016958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:11.017992 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:11.017996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:11.019611 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:11.019621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:11.020530 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:11.020535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:11.020755 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:11.020829 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 07:17:11.020910 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 07:17:11.021066 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":384,"spinlock_wait_cycles":7168}
I20260620 07:17:11.021781 19060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc42ae465d849d38dcc4947a5519349" candidate_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dfdc18ed2bc434abea354b7c13b888a" is_pre_election: true
I20260620 07:17:11.022042 19060 raft_consensus.cc:2454] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbb35bc6d8714b7599cba2fce4b653c2 in term 0.
I20260620 07:17:11.022070 18923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc42ae465d849d38dcc4947a5519349" candidate_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" is_pre_election: true
I20260620 07:17:11.022250 18923 raft_consensus.cc:2454] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbb35bc6d8714b7599cba2fce4b653c2 in term 0.
I20260620 07:17:11.022387 18719 leader_election.cc:302] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dfdc18ed2bc434abea354b7c13b888a, cbb35bc6d8714b7599cba2fce4b653c2; no voters:
I20260620 07:17:11.022724 19208 raft_consensus.cc:2781] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:17:11.022780 19208 raft_consensus.cc:482] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:17:11.022806 19208 raft_consensus.cc:3037] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:11.024053 19208 raft_consensus.cc:504] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:11.024410 19208 leader_election.cc:288] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [CANDIDATE]: Term 1 election: Requested vote from peers 1dfdc18ed2bc434abea354b7c13b888a (127.11.123.131:36377), 1a5acc88ff0c4c86a67e6b97ea1cee0c (127.11.123.130:42833)
I20260620 07:17:11.024716 19060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc42ae465d849d38dcc4947a5519349" candidate_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dfdc18ed2bc434abea354b7c13b888a"
I20260620 07:17:11.024739 18923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc42ae465d849d38dcc4947a5519349" candidate_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c"
I20260620 07:17:11.024845 19060 raft_consensus.cc:3037] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:11.024860 18923 raft_consensus.cc:3037] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:11.025770 18923 raft_consensus.cc:2454] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbb35bc6d8714b7599cba2fce4b653c2 in term 1.
I20260620 07:17:11.025883 19060 raft_consensus.cc:2454] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbb35bc6d8714b7599cba2fce4b653c2 in term 1.
I20260620 07:17:11.026046 18720 leader_election.cc:302] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a5acc88ff0c4c86a67e6b97ea1cee0c, cbb35bc6d8714b7599cba2fce4b653c2; no voters:
I20260620 07:17:11.026249 19208 raft_consensus.cc:2781] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:11.026473 19208 raft_consensus.cc:686] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 1 LEADER]: Becoming Leader. State: Replica: cbb35bc6d8714b7599cba2fce4b653c2, State: Running, Role: LEADER
I20260620 07:17:11.026742 19208 consensus_queue.cc:237] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } }
I20260620 07:17:11.027138 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.026924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.129:38409)
0620 07:17:11.027017 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.027073 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 07:17:11.029222 18645 catalog_manager.cc:5495] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbb35bc6d8714b7599cba2fce4b653c2 (127.11.123.129). New cstate: current_term: 1 leader_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbb35bc6d8714b7599cba2fce4b653c2" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 } health_report { overall_health: UNKNOWN } } }
I20260620 07:17:11.041486 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.037846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.11.123.129:41141)
0620 07:17:11.037968 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:11.037974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:11.037990 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:17:11.038179 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:11.038182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:11.038262 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:11.038373 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:17:11.038648 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:11.039175 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:11.040051 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:11.040278 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:11.040384 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:11.040648 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:17:11.041070 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:17:11.041234 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:11.041294 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 07:17:11.041344 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:17:11.043865 18923 raft_consensus.cc:1261] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 1 FOLLOWER]: Refusing update from remote peer cbb35bc6d8714b7599cba2fce4b653c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:11.044072 19060 raft_consensus.cc:1261] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a [term 1 FOLLOWER]: Refusing update from remote peer cbb35bc6d8714b7599cba2fce4b653c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:11.044339 19208 consensus_queue.cc:1035] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a5acc88ff0c4c86a67e6b97ea1cee0c" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:11.044629 19212 consensus_queue.cc:1035] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dfdc18ed2bc434abea354b7c13b888a" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:11.047048 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.046761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.131:50931)
0620 07:17:11.046896 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.046966 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 07:17:11.047073 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.046678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.130:38321)
0620 07:17:11.046914 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.046977 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260620 07:17:11.049700 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.049405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.130:43653)
0620 07:17:11.049585 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.049637 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260620 07:17:11.050931 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.048814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.129:40257)
0620 07:17:11.050814 (+ 2000us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.050866 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1936}
I20260620 07:17:11.053640 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.053283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.131:41373)
0620 07:17:11.053529 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.053576 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260620 07:17:11.054277 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.053575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46763 (local address 127.11.123.129:36805)
0620 07:17:11.054170 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.054220 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534}
I20260620 07:17:11.056452 18645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:11.056600 18645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:11.060317 18645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d36ebdb692ec4976980191ef158e3e29: Sending DeleteTablet for 3 replicas of tablet 6cc42ae465d849d38dcc4947a5519349
I20260620 07:17:11.060892 19039 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cc42ae465d849d38dcc4947a5519349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:11 UTC) from {username='oryx', principal='oryx/127.11.123.188@KRBTEST.COM'} at 127.0.0.1:46984
I20260620 07:17:11.061236 18903 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cc42ae465d849d38dcc4947a5519349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:11 UTC) from {username='oryx', principal='oryx/127.11.123.188@KRBTEST.COM'} at 127.0.0.1:33716
I20260620 07:17:11.061578 19226 tablet_replica.cc:330] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: stopping tablet replica
I20260620 07:17:11.061568 18766 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cc42ae465d849d38dcc4947a5519349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:11 UTC) from {username='oryx', principal='oryx/127.11.123.188@KRBTEST.COM'} at 127.0.0.1:33110
I20260620 07:17:11.061705 19226 raft_consensus.cc:2229] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:11.061851 19226 raft_consensus.cc:2258] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:11.061967 19227 tablet_replica.cc:330] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2: stopping tablet replica
I20260620 07:17:11.062045 19225 tablet_replica.cc:330] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a: stopping tablet replica
I20260620 07:17:11.062155 19225 raft_consensus.cc:2229] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:11.062163 19227 raft_consensus.cc:2229] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:17:11.062273 19225 raft_consensus.cc:2258] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:11.062549 19227 raft_consensus.cc:2258] T 6cc42ae465d849d38dcc4947a5519349 P cbb35bc6d8714b7599cba2fce4b653c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:11.062597 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 18614
I20260620 07:17:11.064900 19225 ts_tablet_manager.cc:1905] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:11.068377 19225 ts_tablet_manager.cc:1918] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:11.068495 19225 log.cc:1196] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/6cc42ae465d849d38dcc4947a5519349
I20260620 07:17:11.068826 19225 ts_tablet_manager.cc:1939] T 6cc42ae465d849d38dcc4947a5519349 P 1dfdc18ed2bc434abea354b7c13b888a: Deleting consensus metadata
I20260620 07:17:11.069664 19226 ts_tablet_manager.cc:1905] T 6cc42ae465d849d38dcc4947a5519349 P 1a5acc88ff0c4c86a67e6b97ea1cee0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:11.069629 18637 catalog_manager.cc:4841] TS 1dfdc18ed2bc434abea354b7c13b888a (127.11.123.131:36377): tablet 6cc42ae465d849d38dcc4947a5519349 (table test-table [id=dc215a4b78e1479ca16948812b839298]) successfully deleted
I20260620 07:17:11.071367 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 18840
I20260620 07:17:11.071516 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.071239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:41141 (local address 127.0.0.1:33132)
0620 07:17:11.071346 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.071395 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 07:17:11.073280 18640 catalog_manager.cc:4584] TS cbb35bc6d8714b7599cba2fce4b653c2 (127.11.123.129:41141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cc42ae465d849d38dcc4947a5519349: Network error: Client connection negotiation failed: client connection to 127.11.123.129:41141: connect: Connection refused (error 111)
I20260620 07:17:11.077696 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 18977
I20260620 07:17:11.078088 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:11.077769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:42833 (local address 127.0.0.1:33720)
0620 07:17:11.077950 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:11.078008 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.130:42833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 07:17:11.084507 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 18332
I20260620 07:17:11.090770 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 18441
2026-06-20T07:17:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5292 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:17:11 dist-test-slave-j663 krb5kdc[19230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:17:11 dist-test-slave-j663 krb5kdc[19230](info): set up 2 sockets
Jun 20 07:17:11 dist-test-slave-j663 krb5kdc[19230](info): commencing operation
krb5kdc: starting...
W20260620 07:17:12.125547 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:17:12 dist-test-slave-j663 krb5kdc[19230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:17:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.190.keytab.
Entry for principal oryx/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.190.keytab.
I20260620 07:17:12.243988 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:42677
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.190.keytab
--principal=oryx/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:42677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:12.351078 19243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:12.351306 19243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:12.351379 19243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:12.355338 19243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:12.355388 19243 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 07:17:12.355406 19243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:12.355466 19243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:12.355501 19243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:12.355541 19243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:17:12.360735 19243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:42677
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.190.keytab
--principal=oryx/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:42677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:12.361739 19243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:12.362721 19243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:12.370523 19251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:12.370497 19248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:12.370494 19249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:12.370690 19243 server_base.cc:1034] running on GCE node
I20260620 07:17:12.371276 19243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:12.372326 19243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:12.373476 19243 hybrid_clock.cc:648] HybridClock initialized: now 1781939832373439 us; error 50 us; skew 500 ppm
Jun 20 07:17:12 dist-test-slave-j663 krb5kdc[19230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939832, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:12.376696 19243 init.cc:377] Logged in from keytab as oryx/127.11.123.190@KRBTEST.COM (short username oryx)
I20260620 07:17:12.377663 19243 webserver.cc:460] Webserver started at http://127.11.123.190:34393/ using document root <none> and password file <none>
I20260620 07:17:12.378212 19243 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:12.378301 19243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:12.378502 19243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:12.380533 19243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "8f02ab269a6048dab9eb3976b5f6f005"
format_stamp: "Formatted at 2026-06-20 07:17:12 on dist-test-slave-j663"
server_key: "a071183a16d40b4df677478c6d18a8b0"
server_key_iv: "dd5bde30fd02113b65b57cacbfc44585"
server_key_version: "encryptionkey@0"
I20260620 07:17:12.381120 19243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "8f02ab269a6048dab9eb3976b5f6f005"
format_stamp: "Formatted at 2026-06-20 07:17:12 on dist-test-slave-j663"
server_key: "a071183a16d40b4df677478c6d18a8b0"
server_key_iv: "dd5bde30fd02113b65b57cacbfc44585"
server_key_version: "encryptionkey@0"
I20260620 07:17:12.384545 19243 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:17:12.386830 19258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:12.388031 19243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:12.388182 19243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "8f02ab269a6048dab9eb3976b5f6f005"
format_stamp: "Formatted at 2026-06-20 07:17:12 on dist-test-slave-j663"
server_key: "a071183a16d40b4df677478c6d18a8b0"
server_key_iv: "dd5bde30fd02113b65b57cacbfc44585"
server_key_version: "encryptionkey@0"
I20260620 07:17:12.388289 19243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:12.405766 19243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:12.408882 19243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:12.409111 19243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:12.416889 19243 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:42677
I20260620 07:17:12.416913 19320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:42677 every 8 connection(s)
I20260620 07:17:12.417820 19243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:12.419528 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 19243
I20260620 07:17:12.419637 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:17:12.419965 11758 external_mini_cluster.cc:1428] Setting key 8a5b32103cfe2167dc5d6da64732829a
I20260620 07:17:12.420742 19321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:12.427690 19321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005: Bootstrap starting.
Jun 20 07:17:12 dist-test-slave-j663 krb5kdc[19230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:12.430132 19321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:12.431504 19321 log.cc:824] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:12.433708 19321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005: No bootstrap required, opened a new log
I20260620 07:17:12.435063 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:12.421387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.11.123.190:42677)
0620 07:17:12.422074 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:12.422094 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:12.422139 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:17:12.422989 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:12.423012 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:12.423425 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:12.423826 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:12.424444 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.425924 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.426776 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.427419 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.427574 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:12.430163 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:12.430169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:12.430185 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:12.430217 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:12.432130 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:12.432557 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:12.432563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:12.432568 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:12.432707 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:12.432955 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:12.432959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:12.432960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:12.433455 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:12.433655 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:12.433830 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 07:17:12.434119 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":128,"threads_started":1}
I20260620 07:17:12.437327 19321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f02ab269a6048dab9eb3976b5f6f005" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 42677 } }
I20260620 07:17:12.437527 19321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:12.437561 19321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f02ab269a6048dab9eb3976b5f6f005, State: Initialized, Role: FOLLOWER
I20260620 07:17:12.437897 19321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f02ab269a6048dab9eb3976b5f6f005" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 42677 } }
I20260620 07:17:12.438001 19321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:12.438045 19321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:12.438118 19321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:12.438889 19321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f02ab269a6048dab9eb3976b5f6f005" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 42677 } }
I20260620 07:17:12.439179 19321 leader_election.cc:302] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f02ab269a6048dab9eb3976b5f6f005; no voters:
I20260620 07:17:12.439416 19321 leader_election.cc:288] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:17:12.439535 19326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:12.440021 19326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [term 1 LEADER]: Becoming Leader. State: Replica: 8f02ab269a6048dab9eb3976b5f6f005, State: Running, Role: LEADER
I20260620 07:17:12.440310 19326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f02ab269a6048dab9eb3976b5f6f005" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 42677 } }
I20260620 07:17:12.440661 19321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:12.441993 19327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f02ab269a6048dab9eb3976b5f6f005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f02ab269a6048dab9eb3976b5f6f005" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 42677 } } }
I20260620 07:17:12.442133 19327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:12.443190 19328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f02ab269a6048dab9eb3976b5f6f005. Latest consensus state: current_term: 1 leader_uuid: "8f02ab269a6048dab9eb3976b5f6f005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f02ab269a6048dab9eb3976b5f6f005" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 42677 } } }
I20260620 07:17:12.443542 19328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005 [sys.catalog]: This master's current role is: LEADER
W20260620 07:17:12.445597 19342 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:17:12.445660 19342 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:17:12.445737 19336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:12.446426 19336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:12.451476 19336 catalog_manager.cc:1347] Generated new cluster ID: 43a1b5026f6d43c5b9e8fa689908a003
I20260620 07:17:12.451550 19336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:12.457583 19336 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:17:12.458427 19336 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:12.467161 19336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005: Generated new TSK 0
I20260620 07:17:12.467820 19336 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab.
Entry for principal oryx/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab.
I20260620 07:17:12.503645 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:42677
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab
--principal=oryx/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:12.612505 19347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:12.612741 19347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:12.612782 19347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:12.617256 19347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:12.617316 19347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:12.617353 19347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:17:12.617460 19347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:17:12.622610 19347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab
--principal=oryx/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.123.190:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:12.623607 19347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:12.624594 19347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:12.633785 19353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:12.633785 19352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:12.633832 19355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:12.633832 19347 server_base.cc:1034] running on GCE node
I20260620 07:17:12.634416 19347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:12.635027 19347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:12.636183 19347 hybrid_clock.cc:648] HybridClock initialized: now 1781939832636156 us; error 37 us; skew 500 ppm
Jun 20 07:17:12 dist-test-slave-j663 krb5kdc[19230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939832, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:12.638808 19347 init.cc:377] Logged in from keytab as oryx/127.11.123.129@KRBTEST.COM (short username oryx)
I20260620 07:17:12.639683 19347 webserver.cc:460] Webserver started at http://127.11.123.129:39727/ using document root <none> and password file <none>
I20260620 07:17:12.640271 19347 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:12.640332 19347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:12.640554 19347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:12.642688 19347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "3d49888c09ec4fb19d37486c92432902"
format_stamp: "Formatted at 2026-06-20 07:17:12 on dist-test-slave-j663"
server_key: "f6acd69466832c9fdf2d2bb6b271c70b"
server_key_iv: "4979923963426a405b9453f7a2a16872"
server_key_version: "encryptionkey@0"
I20260620 07:17:12.643265 19347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "3d49888c09ec4fb19d37486c92432902"
format_stamp: "Formatted at 2026-06-20 07:17:12 on dist-test-slave-j663"
server_key: "f6acd69466832c9fdf2d2bb6b271c70b"
server_key_iv: "4979923963426a405b9453f7a2a16872"
server_key_version: "encryptionkey@0"
I20260620 07:17:12.646728 19347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:17:12.648912 19362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:12.650048 19347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:12.650163 19347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "3d49888c09ec4fb19d37486c92432902"
format_stamp: "Formatted at 2026-06-20 07:17:12 on dist-test-slave-j663"
server_key: "f6acd69466832c9fdf2d2bb6b271c70b"
server_key_iv: "4979923963426a405b9453f7a2a16872"
server_key_version: "encryptionkey@0"
I20260620 07:17:12.650261 19347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:12.667589 19347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:12.670172 19347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:12.670358 19347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:12.671695 19347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:12.671766 19347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:12.671833 19347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:12.671883 19347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:17:12 dist-test-slave-j663 krb5kdc[19230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939832, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:12.682590 19347 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:45719
I20260620 07:17:12.683521 19347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:17:12.684291 19477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:45719 every 8 connection(s)
I20260620 07:17:12.685271 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:12.672328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:43969 (local address 127.11.123.190:42677)
0620 07:17:12.672468 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:12.672473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:12.673336 (+ 863us) server_negotiation.cc:407] Connection header received
0620 07:17:12.674222 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:12.674226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:12.674282 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:12.674381 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:12.675694 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.676581 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.677921 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.678336 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.678467 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:12.681461 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:12.681465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:12.681468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:12.681500 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:12.683043 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:12.683622 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:12.683625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:12.683627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:12.683732 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:12.684115 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:12.684119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:12.684120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:12.684330 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:12.684426 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:12.684978 (+ 552us) server_negotiation.cc:299] Negotiation successful
0620 07:17:12.685131 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:17:12.685942 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:12.672726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:42677 (local address 127.11.123.129:43969)
0620 07:17:12.673178 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:12.673225 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:12.674022 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:12.674747 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:12.674755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:12.675067 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:12.675495 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:12.675507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.676715 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.676718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:12.677819 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:12.677828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.678509 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.678515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:12.678712 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:12.679342 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:12.679369 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:12.681264 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:12.683148 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:12.683155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:12.683170 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:12.683490 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:12.683821 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:12.683825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:12.683827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:12.684003 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:12.684476 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:12.684481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:12.684742 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 07:17:12.684884 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 07:17:12.685169 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":133,"threads_started":1}
I20260620 07:17:12.687206 19478 heartbeater.cc:347] Connected to a master server at 127.11.123.190:42677
I20260620 07:17:12.687350 19478 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:12.687809 19478 heartbeater.cc:511] Master 127.11.123.190:42677 requested a full tablet report, sending...
I20260620 07:17:12.689397 19275 ts_manager.cc:194] Registered new tserver with Master: 3d49888c09ec4fb19d37486c92432902 (127.11.123.129:45719)
I20260620 07:17:12.689543 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 19347
I20260620 07:17:12.689678 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:17:12.689985 11758 external_mini_cluster.cc:1428] Setting key dc86fcbe4ca906b5f507019c985bed21
I20260620 07:17:12.691000 19275 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:43969
I20260620 07:17:12.697892 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:12.689348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:52141 (local address 127.11.123.190:42677)
0620 07:17:12.689465 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:12.689471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:12.690524 (+ 1053us) server_negotiation.cc:407] Connection header received
0620 07:17:12.690662 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:12.690664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:12.690705 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:12.690788 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:12.691337 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.692113 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.693195 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.693573 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.693687 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:12.694841 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:12.694844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:12.694846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:12.694866 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:12.696282 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:12.696777 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:12.696779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:12.696780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:12.696844 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:12.697189 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:12.697192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:12.697193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:12.697334 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:12.697407 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:12.697712 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 07:17:12.697792 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:17:12.698045 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:12.689238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:42677 (local address 127.11.123.129:52141)
0620 07:17:12.690442 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:12.690456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:12.690555 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:12.690818 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:12.690821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:12.691024 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:12.691189 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:12.691197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.692262 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.692265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:12.693068 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:12.693100 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.693687 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.693689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:12.693755 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:12.694252 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:12.694271 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:12.694710 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:12.696394 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:12.696401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:12.696406 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:12.696679 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:12.696936 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:12.696942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:12.696946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:12.697061 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:12.697433 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:12.697439 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:12.697545 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 07:17:12.697666 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 07:17:12.697809 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for oryx/127.11.123.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab.
Entry for principal oryx/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab.
I20260620 07:17:12.716655 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:42677
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab
--principal=oryx/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:12.820075 19487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:12.820353 19487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:12.820428 19487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:12.824321 19487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:12.824409 19487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:12.824478 19487 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:17:12.824566 19487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:17:12.829489 19487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab
--principal=oryx/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.123.190:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:12.830466 19487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:12.831404 19487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:12.840431 19495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:12.840487 19487 server_base.cc:1034] running on GCE node
W20260620 07:17:12.840443 19492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:12.840447 19493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:12.841063 19487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:12.841807 19487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:12.842983 19487 hybrid_clock.cc:648] HybridClock initialized: now 1781939832842974 us; error 33 us; skew 500 ppm
Jun 20 07:17:12 dist-test-slave-j663 krb5kdc[19230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939832, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:12.845500 19487 init.cc:377] Logged in from keytab as oryx/127.11.123.130@KRBTEST.COM (short username oryx)
I20260620 07:17:12.846330 19487 webserver.cc:460] Webserver started at http://127.11.123.130:44545/ using document root <none> and password file <none>
I20260620 07:17:12.846854 19487 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:12.846941 19487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:12.847177 19487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:12.849037 19487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "485c95b97cdf45db8f4ca32c73bef46e"
format_stamp: "Formatted at 2026-06-20 07:17:12 on dist-test-slave-j663"
server_key: "41a0fb634b54eb52a171d83e9b77dc7c"
server_key_iv: "3c172673beb7b9516db7a1c1e2653129"
server_key_version: "encryptionkey@0"
I20260620 07:17:12.849635 19487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "485c95b97cdf45db8f4ca32c73bef46e"
format_stamp: "Formatted at 2026-06-20 07:17:12 on dist-test-slave-j663"
server_key: "41a0fb634b54eb52a171d83e9b77dc7c"
server_key_iv: "3c172673beb7b9516db7a1c1e2653129"
server_key_version: "encryptionkey@0"
I20260620 07:17:12.853006 19487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:17:12.855270 19502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:12.856376 19487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:17:12.856501 19487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "485c95b97cdf45db8f4ca32c73bef46e"
format_stamp: "Formatted at 2026-06-20 07:17:12 on dist-test-slave-j663"
server_key: "41a0fb634b54eb52a171d83e9b77dc7c"
server_key_iv: "3c172673beb7b9516db7a1c1e2653129"
server_key_version: "encryptionkey@0"
I20260620 07:17:12.856604 19487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:12.867542 19487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:12.870294 19487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:12.870483 19487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:12.871829 19487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:12.871898 19487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:12.871965 19487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:12.872018 19487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:17:12 dist-test-slave-j663 krb5kdc[19230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939832, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:12.882912 19487 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:43395
I20260620 07:17:12.882989 19617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:43395 every 8 connection(s)
I20260620 07:17:12.883917 19487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:17:12.885388 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:12.872567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:40383 (local address 127.11.123.190:42677)
0620 07:17:12.872726 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:12.872733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:12.873616 (+ 883us) server_negotiation.cc:407] Connection header received
0620 07:17:12.874895 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:12.874899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:12.874962 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:12.875083 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:12.876325 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.876961 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.878180 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.878397 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.878511 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:12.881596 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:12.881606 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:12.881608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:12.881656 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:12.883255 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:12.883819 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:12.883821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:12.883822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:12.883893 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:12.884278 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:12.884282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:12.884286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:12.884508 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:12.884604 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:12.885163 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 07:17:12.885274 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:17:12.886087 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:12.872938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:42677 (local address 127.11.123.130:40383)
0620 07:17:12.873441 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:12.873496 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:12.874606 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:12.875323 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:12.875334 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:12.875695 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:12.876188 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:12.876200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.877098 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.877104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:12.878058 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:12.878067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.878536 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.878541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:12.878745 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:12.879294 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:12.879316 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:12.881435 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:12.883372 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:12.883378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:12.883389 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:12.883716 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:12.883981 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:12.883985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:12.883986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:12.884155 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:12.884691 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:12.884696 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:12.884897 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 07:17:12.885034 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 07:17:12.885306 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":152,"threads_started":1}
I20260620 07:17:12.887254 19618 heartbeater.cc:347] Connected to a master server at 127.11.123.190:42677
I20260620 07:17:12.887399 19618 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:12.887916 19618 heartbeater.cc:511] Master 127.11.123.190:42677 requested a full tablet report, sending...
I20260620 07:17:12.888901 19275 ts_manager.cc:194] Registered new tserver with Master: 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130:43395)
I20260620 07:17:12.889660 19275 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:40383
I20260620 07:17:12.892009 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 19487
I20260620 07:17:12.892138 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:17:12.892522 11758 external_mini_cluster.cc:1428] Setting key 6b8ad149617ec1788b5bf214b15df656
I20260620 07:17:12.896517 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:12.889369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:42677 (local address 127.11.123.130:45131)
0620 07:17:12.889643 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:12.889658 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:12.889738 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:12.890034 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:12.890037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:12.890231 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:12.890346 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:12.890353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.891043 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.891047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:12.891838 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:12.891849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.892342 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.892346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:12.892443 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:12.892904 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:12.892920 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:12.893339 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:12.894961 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:12.894967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:12.894972 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:12.895235 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:12.895554 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:12.895557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:12.895559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:12.895644 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:12.896116 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:12.896118 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:12.896184 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 07:17:12.896261 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 07:17:12.896352 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":105,"threads_started":1}
I20260620 07:17:12.896660 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:12.889466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:45131 (local address 127.11.123.190:42677)
0620 07:17:12.889587 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:12.889591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:12.889729 (+ 138us) server_negotiation.cc:407] Connection header received
0620 07:17:12.889869 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:12.889874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:12.889918 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:12.890002 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:12.890456 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.890915 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.891945 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:12.892211 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:12.892319 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:12.893457 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:12.893460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:12.893463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:12.893483 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:12.894852 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:12.895343 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:12.895348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:12.895351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:12.895448 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:12.895731 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:12.895736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:12.895739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:12.896020 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:12.896127 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:12.896342 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 07:17:12.896472 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for oryx/127.11.123.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab.
Entry for principal oryx/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab.
I20260620 07:17:12.920635 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:42677
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab
--principal=oryx/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:13.030407 19627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:13.030663 19627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:13.030704 19627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:13.034629 19627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:13.034685 19627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:13.034722 19627 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:17:13.034754 19627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:17:13.040036 19627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab
--principal=oryx/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.123.190:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:13.041457 19627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:13.042510 19627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:13.051823 19633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:13.051792 19632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:13.051800 19635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:13.051928 19627 server_base.cc:1034] running on GCE node
I20260620 07:17:13.052421 19627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:13.053117 19627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:13.054308 19627 hybrid_clock.cc:648] HybridClock initialized: now 1781939833054289 us; error 26 us; skew 500 ppm
Jun 20 07:17:13 dist-test-slave-j663 krb5kdc[19230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939833, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:13.056974 19627 init.cc:377] Logged in from keytab as oryx/127.11.123.131@KRBTEST.COM (short username oryx)
I20260620 07:17:13.057814 19627 webserver.cc:460] Webserver started at http://127.11.123.131:40145/ using document root <none> and password file <none>
I20260620 07:17:13.058301 19627 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:13.058357 19627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:13.058508 19627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:13.060361 19627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "4f07a155a67c48ec8ca225d6a2438676"
format_stamp: "Formatted at 2026-06-20 07:17:13 on dist-test-slave-j663"
server_key: "866b52d8364f2819569a6441dfade24a"
server_key_iv: "f35d9189e8bbcbb50fd3f505fa085b5e"
server_key_version: "encryptionkey@0"
I20260620 07:17:13.060883 19627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "4f07a155a67c48ec8ca225d6a2438676"
format_stamp: "Formatted at 2026-06-20 07:17:13 on dist-test-slave-j663"
server_key: "866b52d8364f2819569a6441dfade24a"
server_key_iv: "f35d9189e8bbcbb50fd3f505fa085b5e"
server_key_version: "encryptionkey@0"
I20260620 07:17:13.064304 19627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:17:13.066488 19642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:13.067582 19627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:13.067677 19627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "4f07a155a67c48ec8ca225d6a2438676"
format_stamp: "Formatted at 2026-06-20 07:17:13 on dist-test-slave-j663"
server_key: "866b52d8364f2819569a6441dfade24a"
server_key_iv: "f35d9189e8bbcbb50fd3f505fa085b5e"
server_key_version: "encryptionkey@0"
I20260620 07:17:13.067751 19627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:13.094393 19627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:13.097237 19627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:13.097393 19627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:13.098774 19627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:13.098822 19627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:13.098860 19627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:13.098876 19627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:17:13 dist-test-slave-j663 krb5kdc[19230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939833, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:13.109760 19627 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:37539
I20260620 07:17:13.109813 19757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:37539 every 8 connection(s)
I20260620 07:17:13.110666 19627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:17:13.112576 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.099552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:41019 (local address 127.11.123.190:42677)
0620 07:17:13.099701 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.099707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.100677 (+ 970us) server_negotiation.cc:407] Connection header received
0620 07:17:13.101968 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.101973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.102036 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.102160 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:13.103391 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.103948 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.105473 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.105866 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.105995 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.108916 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:13.108920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:13.108922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:13.108948 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:13.110564 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:13.111135 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:13.111140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:13.111143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:13.111239 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:13.111637 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:13.111640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:13.111642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:13.111854 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:13.111979 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.112368 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.112463 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:17:13.113381 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.100002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:42677 (local address 127.11.123.131:41019)
0620 07:17:13.100496 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.100555 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.101738 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.102332 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.102338 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.102774 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:13.103234 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.103246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.104075 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.104078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.105327 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.105336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.106044 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.106050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.106271 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.106900 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:13.106923 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:13.108737 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:13.110677 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:13.110684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:13.110699 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:13.111013 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:13.111352 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:13.111358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:13.111360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:13.111518 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:13.111945 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:13.111950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:13.112167 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.112317 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.112542 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260620 07:17:13.114637 19758 heartbeater.cc:347] Connected to a master server at 127.11.123.190:42677
I20260620 07:17:13.114801 19758 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:13.115373 19758 heartbeater.cc:511] Master 127.11.123.190:42677 requested a full tablet report, sending...
I20260620 07:17:13.116372 19274 ts_manager.cc:194] Registered new tserver with Master: 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539)
I20260620 07:17:13.117107 19274 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:41019
I20260620 07:17:13.117645 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 19627
I20260620 07:17:13.117762 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:17:13.118031 11758 external_mini_cluster.cc:1428] Setting key ac4178f21c6502337cb04e6bf587c860
I20260620 07:17:13.120158 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:17:13.124332 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.117113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:50865 (local address 127.11.123.190:42677)
0620 07:17:13.117213 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.117218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.117440 (+ 222us) server_negotiation.cc:407] Connection header received
0620 07:17:13.117605 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.117608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.117648 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.117736 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:13.118274 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.118867 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.119925 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.120194 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.120304 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.121387 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:13.121390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:13.121392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:13.121421 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:13.122785 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:13.123282 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:13.123284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:13.123286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:13.123359 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:13.123642 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:13.123644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:13.123645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:13.123792 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:13.123870 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.124151 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.124234 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 07:17:13.124620 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.117043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:42677 (local address 127.11.123.131:50865)
0620 07:17:13.117343 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.117358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.117501 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.117832 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.117836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.118051 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:13.118153 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.118160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.119027 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.119030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.119795 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.119806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.120306 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.120309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.120400 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.120870 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:13.120880 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:13.121266 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:13.122909 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:13.122912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:13.122921 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:13.123189 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:13.123453 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:13.123459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:13.123464 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:13.123561 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:13.123880 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:13.123886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:13.123998 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.124118 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.124274 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":132,"threads_started":1}
Jun 20 07:17:13 dist-test-slave-j663 krb5kdc[19230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:17:13 dist-test-slave-j663 krb5kdc[19230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:13 dist-test-slave-j663 krb5kdc[19230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:17:13.144368 19324 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:13.138109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.11.123.190:42677)
0620 07:17:13.138242 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.138245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.138256 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:17:13.138295 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.138297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.138332 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.138445 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:13.139117 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.139618 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.140541 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.140763 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.140876 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.144199 (+ 3323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38158: BlockingRecv error: recv got EOF from 127.0.0.1:38158 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 07:17:13 dist-test-slave-j663 krb5kdc[19230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:13.153153 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.145721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.11.123.190:42677)
0620 07:17:13.145852 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.145855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.145902 (+ 47us) server_negotiation.cc:407] Connection header received
0620 07:17:13.146013 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.146015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.146049 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.146115 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:13.146718 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.147232 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.147953 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.148161 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.148240 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.150417 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:13.150420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:13.150421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:13.150441 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:13.151805 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:13.152150 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:13.152152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:13.152153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:13.152220 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:13.152432 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:13.152434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:13.152435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:13.152581 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:13.152674 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.152841 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.152919 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:17:13.159462 19274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:17:13.162532 19274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:17:13.177150 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.170238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:43395 (local address 127.0.0.1:34808)
0620 07:17:13.171018 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.171067 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.171196 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.172307 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.172313 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.172331 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:13.172564 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.172569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.173971 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.173974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.175517 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.175523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.176566 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.176570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.176693 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.176741 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.176840 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.176971 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":230,"thread_start_us":73,"threads_started":1}
I20260620 07:17:13.177533 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.170599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48986 (local address 127.11.123.129:45719)
0620 07:17:13.171682 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.171698 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.171719 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:17:13.171780 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.171784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.171956 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.172138 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:13.172432 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.173689 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.175635 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.176434 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.176576 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.176702 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.177260 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.177374 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":86,"threads_started":1}
I20260620 07:17:13.177706 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.170383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.11.123.130:43395)
0620 07:17:13.171442 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.171456 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.171497 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:17:13.171751 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.171756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.171975 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.172138 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:13.172688 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.173832 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.175635 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.176434 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.176576 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.176702 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.177484 (+ 782us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.177571 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":101,"threads_started":1}
I20260620 07:17:13.177721 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.171336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.11.123.131:37539)
0620 07:17:13.171878 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.171891 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.171911 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:17:13.171986 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.171991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.172195 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.172348 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:13.173173 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.174418 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.175697 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.176609 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.177314 (+ 705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.177447 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.177490 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.177567 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":101,"threads_started":1}
I20260620 07:17:13.177196 19784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.171134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:37539 (local address 127.0.0.1:41472)
0620 07:17:13.171489 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.171501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.171594 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.172326 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.172329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.172340 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:13.172562 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.172569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.174531 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.174537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.175547 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.175561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.176757 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.176760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.176905 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.176958 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.177015 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.177069 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":149,"thread_start_us":111,"threads_started":1}
I20260620 07:17:13.177806 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.170444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45719 (local address 127.0.0.1:48986)
0620 07:17:13.171018 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.171066 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.171548 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.172138 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.172144 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.172165 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:13.172305 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.172312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.173829 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.173833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.175491 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.175504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.176768 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.176771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.176965 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.176991 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.177060 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.177159 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":167,"thread_start_us":111,"threads_started":1}
I20260620 07:17:13.180495 19692 tablet_service.cc:1467] Processing CreateTablet for tablet 5bdc31f27c99489889ec66096d4f81b9 (DEFAULT_TABLE table=test-table [id=cf98e27645c14299904773db3172adb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:13.180681 19412 tablet_service.cc:1467] Processing CreateTablet for tablet 5bdc31f27c99489889ec66096d4f81b9 (DEFAULT_TABLE table=test-table [id=cf98e27645c14299904773db3172adb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:13.180682 19552 tablet_service.cc:1467] Processing CreateTablet for tablet 5bdc31f27c99489889ec66096d4f81b9 (DEFAULT_TABLE table=test-table [id=cf98e27645c14299904773db3172adb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:13.181542 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bdc31f27c99489889ec66096d4f81b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.181643 19412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bdc31f27c99489889ec66096d4f81b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.181643 19552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bdc31f27c99489889ec66096d4f81b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.187878 19786 tablet_bootstrap.cc:492] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676: Bootstrap starting.
I20260620 07:17:13.189311 19788 tablet_bootstrap.cc:492] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902: Bootstrap starting.
I20260620 07:17:13.189839 19787 tablet_bootstrap.cc:492] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e: Bootstrap starting.
I20260620 07:17:13.190600 19786 tablet_bootstrap.cc:654] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.191370 19788 tablet_bootstrap.cc:654] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.191512 19786 log.cc:824] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:13.191846 19787 tablet_bootstrap.cc:654] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.192232 19788 log.cc:824] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:13.192677 19787 log.cc:824] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:13.193612 19786 tablet_bootstrap.cc:492] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676: No bootstrap required, opened a new log
I20260620 07:17:13.193804 19786 ts_tablet_manager.cc:1397] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:17:13.194223 19788 tablet_bootstrap.cc:492] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902: No bootstrap required, opened a new log
I20260620 07:17:13.194392 19788 ts_tablet_manager.cc:1397] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:17:13.197698 19786 raft_consensus.cc:348] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.197968 19786 raft_consensus.cc:374] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.198032 19786 raft_consensus.cc:729] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f07a155a67c48ec8ca225d6a2438676, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.198028 19788 raft_consensus.cc:348] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.198272 19788 raft_consensus.cc:374] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.198326 19788 raft_consensus.cc:729] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d49888c09ec4fb19d37486c92432902, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.198478 19786 consensus_queue.cc:260] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.198776 19788 consensus_queue.cc:260] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.199137 19758 heartbeater.cc:503] Master 127.11.123.190:42677 was elected leader, sending a full tablet report...
I20260620 07:17:13.199448 19786 ts_tablet_manager.cc:1428] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 07:17:13.199546 19788 ts_tablet_manager.cc:1428] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 07:17:13.199862 19478 heartbeater.cc:503] Master 127.11.123.190:42677 was elected leader, sending a full tablet report...
I20260620 07:17:13.201370 19787 tablet_bootstrap.cc:492] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e: No bootstrap required, opened a new log
I20260620 07:17:13.201571 19787 ts_tablet_manager.cc:1397] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260620 07:17:13.204895 19787 raft_consensus.cc:348] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.205128 19787 raft_consensus.cc:374] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.205185 19787 raft_consensus.cc:729] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 485c95b97cdf45db8f4ca32c73bef46e, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.205562 19787 consensus_queue.cc:260] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.206120 19618 heartbeater.cc:503] Master 127.11.123.190:42677 was elected leader, sending a full tablet report...
I20260620 07:17:13.206398 19787 ts_tablet_manager.cc:1428] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:17:13.214723 19794 raft_consensus.cc:482] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:17:13.214928 19794 raft_consensus.cc:504] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.215943 19794 leader_election.cc:288] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d49888c09ec4fb19d37486c92432902 (127.11.123.129:45719), 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539)
I20260620 07:17:13.219478 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.216148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45719 (local address 127.11.123.130:38727)
0620 07:17:13.216260 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.216276 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.216361 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.216681 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.216684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.216701 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:13.216812 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.216817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.217641 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.217646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.218432 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.218441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.219057 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.219061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.219179 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.219196 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.219285 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.219354 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":18}
I20260620 07:17:13.219477 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.216344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:38727 (local address 127.11.123.129:45719)
0620 07:17:13.216456 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.216460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.216472 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:17:13.216516 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.216518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.216551 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.216640 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:13.216947 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.217530 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.218556 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.218939 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.219066 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.219131 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.219300 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.219363 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:17:13.220099 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.216232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:58395 (local address 127.11.123.131:37539)
0620 07:17:13.216347 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.216353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.216624 (+ 271us) server_negotiation.cc:407] Connection header received
0620 07:17:13.216727 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.216730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.216770 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.216854 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:13.217512 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.218013 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.219027 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.219414 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.219546 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.219607 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.219953 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.220013 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 07:17:13.220137 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.216149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:37539 (local address 127.11.123.130:58395)
0620 07:17:13.216533 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.216544 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.216625 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.216877 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.216880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.216906 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:13.217391 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.217399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.218128 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.218131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.218902 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.218911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.219679 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.219685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.219782 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.219829 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.219927 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.219988 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":121,"thread_start_us":110,"threads_started":1}
I20260620 07:17:13.220083 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bdc31f27c99489889ec66096d4f81b9" candidate_uuid: "485c95b97cdf45db8f4ca32c73bef46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d49888c09ec4fb19d37486c92432902" is_pre_election: true
I20260620 07:17:13.220347 19432 raft_consensus.cc:2454] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 485c95b97cdf45db8f4ca32c73bef46e in term 0.
I20260620 07:17:13.220762 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bdc31f27c99489889ec66096d4f81b9" candidate_uuid: "485c95b97cdf45db8f4ca32c73bef46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f07a155a67c48ec8ca225d6a2438676" is_pre_election: true
I20260620 07:17:13.220983 19503 leader_election.cc:302] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d49888c09ec4fb19d37486c92432902, 485c95b97cdf45db8f4ca32c73bef46e; no voters:
I20260620 07:17:13.221024 19712 raft_consensus.cc:2454] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 485c95b97cdf45db8f4ca32c73bef46e in term 0.
I20260620 07:17:13.221246 19794 raft_consensus.cc:2781] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:17:13.221338 19794 raft_consensus.cc:482] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:17:13.221396 19794 raft_consensus.cc:3037] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:13.222352 19794 raft_consensus.cc:504] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.222646 19794 leader_election.cc:288] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [CANDIDATE]: Term 1 election: Requested vote from peers 3d49888c09ec4fb19d37486c92432902 (127.11.123.129:45719), 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539)
I20260620 07:17:13.222908 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bdc31f27c99489889ec66096d4f81b9" candidate_uuid: "485c95b97cdf45db8f4ca32c73bef46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d49888c09ec4fb19d37486c92432902"
I20260620 07:17:13.222956 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bdc31f27c99489889ec66096d4f81b9" candidate_uuid: "485c95b97cdf45db8f4ca32c73bef46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f07a155a67c48ec8ca225d6a2438676"
I20260620 07:17:13.223052 19432 raft_consensus.cc:3037] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:13.223088 19712 raft_consensus.cc:3037] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:13.223994 19432 raft_consensus.cc:2454] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485c95b97cdf45db8f4ca32c73bef46e in term 1.
I20260620 07:17:13.224023 19712 raft_consensus.cc:2454] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485c95b97cdf45db8f4ca32c73bef46e in term 1.
I20260620 07:17:13.224256 19503 leader_election.cc:302] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d49888c09ec4fb19d37486c92432902, 485c95b97cdf45db8f4ca32c73bef46e; no voters:
I20260620 07:17:13.224408 19794 raft_consensus.cc:2781] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:13.224617 19794 raft_consensus.cc:686] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 LEADER]: Becoming Leader. State: Replica: 485c95b97cdf45db8f4ca32c73bef46e, State: Running, Role: LEADER
I20260620 07:17:13.224910 19794 consensus_queue.cc:237] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.227759 19274 catalog_manager.cc:5495] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e reported cstate change: term changed from 0 to 1, leader changed from <none> to 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130). New cstate: current_term: 1 leader_uuid: "485c95b97cdf45db8f4ca32c73bef46e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } health_report { overall_health: UNKNOWN } } }
I20260620 07:17:13.240244 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.237049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34824 (local address 127.11.123.130:43395)
0620 07:17:13.237200 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.237204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.237221 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:17:13.237361 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.237363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.237408 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.237491 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:17:13.237681 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.238168 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.238906 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.239118 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.239191 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.239442 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:17:13.239812 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:17:13.239923 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.240007 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.240088 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:17:13.245855 19432 raft_consensus.cc:1261] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Refusing update from remote peer 485c95b97cdf45db8f4ca32c73bef46e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:13.245930 19712 raft_consensus.cc:1261] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Refusing update from remote peer 485c95b97cdf45db8f4ca32c73bef46e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:13.246483 19796 consensus_queue.cc:1035] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:13.246685 19794 consensus_queue.cc:1035] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:13.254359 19800 mvcc.cc:205] Tried to move back new op lower bound from 7298825556967976960 to 7298825556891676672. Current Snapshot: MvccSnapshot[applied={T|T < 7298825556967976960}]
I20260620 07:17:13.254990 19799 mvcc.cc:205] Tried to move back new op lower bound from 7298825556967976960 to 7298825556891676672. Current Snapshot: MvccSnapshot[applied={T|T < 7298825556967976960 or (T in {7298825556967976960})}]
I20260620 07:17:13.262001 19274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:13.262181 19274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:13.264629 19274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005: Sending DeleteTablet for 3 replicas of tablet 5bdc31f27c99489889ec66096d4f81b9
I20260620 07:17:13.265364 19552 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bdc31f27c99489889ec66096d4f81b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:13 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:34808
I20260620 07:17:13.265431 19412 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bdc31f27c99489889ec66096d4f81b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:13 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:48986
I20260620 07:17:13.265695 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bdc31f27c99489889ec66096d4f81b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:13 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:41472
I20260620 07:17:13.266141 19811 tablet_replica.cc:330] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676: stopping tablet replica
I20260620 07:17:13.266448 19811 raft_consensus.cc:2229] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:13.266533 19809 tablet_replica.cc:330] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e: stopping tablet replica
I20260620 07:17:13.266745 19811 raft_consensus.cc:2258] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:13.266788 19809 raft_consensus.cc:2229] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:17:13.266819 19274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 07:17:13.267119 19809 raft_consensus.cc:2258] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 07:17:13.267195 19274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:17:13.267470 19810 tablet_replica.cc:330] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902: stopping tablet replica
I20260620 07:17:13.267747 19810 raft_consensus.cc:2229] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:13.268060 19810 raft_consensus.cc:2258] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:13.268524 19809 ts_tablet_manager.cc:1905] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:13.270116 19811 ts_tablet_manager.cc:1905] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:13.271608 19809 ts_tablet_manager.cc:1918] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:13.271770 19809 log.cc:1196] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/5bdc31f27c99489889ec66096d4f81b9
I20260620 07:17:13.272212 19809 ts_tablet_manager.cc:1939] T 5bdc31f27c99489889ec66096d4f81b9 P 485c95b97cdf45db8f4ca32c73bef46e: Deleting consensus metadata
I20260620 07:17:13.273397 19811 ts_tablet_manager.cc:1918] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:13.273422 19261 catalog_manager.cc:4841] TS 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130:43395): tablet 5bdc31f27c99489889ec66096d4f81b9 (table test-table [id=cf98e27645c14299904773db3172adb6]) successfully deleted
I20260620 07:17:13.273447 19412 tablet_service.cc:1467] Processing CreateTablet for tablet 092781a6e06a45798073b595ba982674 (DEFAULT_TABLE table=test-table [id=884a0df63ab64c23ab5d105025c1c95b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:13.273645 19811 log.cc:1196] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/5bdc31f27c99489889ec66096d4f81b9
I20260620 07:17:13.273741 19412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092781a6e06a45798073b595ba982674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.273964 19811 ts_tablet_manager.cc:1939] T 5bdc31f27c99489889ec66096d4f81b9 P 4f07a155a67c48ec8ca225d6a2438676: Deleting consensus metadata
I20260620 07:17:13.275151 19262 catalog_manager.cc:4841] TS 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539): tablet 5bdc31f27c99489889ec66096d4f81b9 (table test-table [id=cf98e27645c14299904773db3172adb6]) successfully deleted
I20260620 07:17:13.276001 19788 tablet_bootstrap.cc:492] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902: Bootstrap starting.
I20260620 07:17:13.276228 19552 tablet_service.cc:1467] Processing CreateTablet for tablet 092781a6e06a45798073b595ba982674 (DEFAULT_TABLE table=test-table [id=884a0df63ab64c23ab5d105025c1c95b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:13.276479 19552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092781a6e06a45798073b595ba982674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.276984 19788 tablet_bootstrap.cc:654] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.278302 19788 tablet_bootstrap.cc:492] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902: No bootstrap required, opened a new log
I20260620 07:17:13.278398 19788 ts_tablet_manager.cc:1397] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:13.278800 19787 tablet_bootstrap.cc:492] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e: Bootstrap starting.
I20260620 07:17:13.278872 19788 raft_consensus.cc:348] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.278978 19788 raft_consensus.cc:374] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.279011 19788 raft_consensus.cc:729] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d49888c09ec4fb19d37486c92432902, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.279007 19692 tablet_service.cc:1467] Processing CreateTablet for tablet 092781a6e06a45798073b595ba982674 (DEFAULT_TABLE table=test-table [id=884a0df63ab64c23ab5d105025c1c95b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:13.279131 19788 consensus_queue.cc:260] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.279255 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092781a6e06a45798073b595ba982674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.279687 19787 tablet_bootstrap.cc:654] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.279985 19788 ts_tablet_manager.cc:1428] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:13.280929 19787 tablet_bootstrap.cc:492] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e: No bootstrap required, opened a new log
I20260620 07:17:13.281002 19787 ts_tablet_manager.cc:1397] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:13.281415 19787 raft_consensus.cc:348] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.281507 19787 raft_consensus.cc:374] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.281535 19787 raft_consensus.cc:729] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 485c95b97cdf45db8f4ca32c73bef46e, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.281641 19787 consensus_queue.cc:260] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.281877 19787 ts_tablet_manager.cc:1428] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:17:13.282064 19786 tablet_bootstrap.cc:492] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676: Bootstrap starting.
I20260620 07:17:13.282929 19786 tablet_bootstrap.cc:654] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.284128 19786 tablet_bootstrap.cc:492] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676: No bootstrap required, opened a new log
I20260620 07:17:13.284202 19786 ts_tablet_manager.cc:1397] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:13.284595 19786 raft_consensus.cc:348] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.284689 19786 raft_consensus.cc:374] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.284722 19786 raft_consensus.cc:729] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f07a155a67c48ec8ca225d6a2438676, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.284828 19786 consensus_queue.cc:260] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.285059 19786 ts_tablet_manager.cc:1428] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:17:13.285375 19810 ts_tablet_manager.cc:1905] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:13.288676 19810 ts_tablet_manager.cc:1918] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:13.288753 19810 log.cc:1196] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/wal/wals/5bdc31f27c99489889ec66096d4f81b9
I20260620 07:17:13.289115 19810 ts_tablet_manager.cc:1939] T 5bdc31f27c99489889ec66096d4f81b9 P 3d49888c09ec4fb19d37486c92432902: Deleting consensus metadata
I20260620 07:17:13.290287 19259 catalog_manager.cc:4841] TS 3d49888c09ec4fb19d37486c92432902 (127.11.123.129:45719): tablet 5bdc31f27c99489889ec66096d4f81b9 (table test-table [id=cf98e27645c14299904773db3172adb6]) successfully deleted
I20260620 07:17:13.353582 19793 raft_consensus.cc:482] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:17:13.353714 19793 raft_consensus.cc:504] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.354853 19793 leader_election.cc:288] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130:43395), 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539)
I20260620 07:17:13.359455 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.355454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:49593 (local address 127.11.123.131:37539)
0620 07:17:13.355570 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.355574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.355589 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:17:13.355689 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.355692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.355742 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.355829 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:13.356140 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.356917 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.358371 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.358786 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.358927 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.358989 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.359205 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.359285 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:17:13.359584 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.355068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:43395 (local address 127.11.123.129:57341)
0620 07:17:13.355461 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.355476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.355597 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.355981 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.355984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.355996 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:13.356098 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.356105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.356805 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.356809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.358194 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.358206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.359157 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.359161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.359255 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.359275 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.359343 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.359413 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":24,"thread_start_us":68,"threads_started":1}
I20260620 07:17:13.359586 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.355204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:57341 (local address 127.11.123.130:43395)
0620 07:17:13.355344 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.355350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.355546 (+ 196us) server_negotiation.cc:407] Connection header received
0620 07:17:13.355745 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.355748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.355787 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.355880 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:13.356191 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.356682 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.358342 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.358998 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.359154 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.359203 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.359378 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.359461 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:17:13.360157 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.355158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:37539 (local address 127.11.123.129:49593)
0620 07:17:13.355461 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.355476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.355593 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.355856 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.355860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.355881 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:13.356011 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.356018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.357034 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.357037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.358205 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.358211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.358920 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.358923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.359037 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.359062 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.359937 (+ 875us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.360025 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":40,"thread_start_us":102,"threads_started":1}
I20260620 07:17:13.360409 19572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092781a6e06a45798073b595ba982674" candidate_uuid: "3d49888c09ec4fb19d37486c92432902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485c95b97cdf45db8f4ca32c73bef46e" is_pre_election: true
I20260620 07:17:13.360639 19572 raft_consensus.cc:2454] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d49888c09ec4fb19d37486c92432902 in term 0.
I20260620 07:17:13.360689 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092781a6e06a45798073b595ba982674" candidate_uuid: "3d49888c09ec4fb19d37486c92432902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f07a155a67c48ec8ca225d6a2438676" is_pre_election: true
I20260620 07:17:13.360839 19712 raft_consensus.cc:2454] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d49888c09ec4fb19d37486c92432902 in term 0.
I20260620 07:17:13.361135 19365 leader_election.cc:302] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d49888c09ec4fb19d37486c92432902, 485c95b97cdf45db8f4ca32c73bef46e; no voters:
I20260620 07:17:13.361416 19793 raft_consensus.cc:2781] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:17:13.361490 19793 raft_consensus.cc:482] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:17:13.361536 19793 raft_consensus.cc:3037] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Advancing to term 1
W20260620 07:17:13.361747 19759 tablet.cc:2369] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:17:13.362583 19793 raft_consensus.cc:504] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.362954 19793 leader_election.cc:288] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [CANDIDATE]: Term 1 election: Requested vote from peers 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130:43395), 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539)
I20260620 07:17:13.363281 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092781a6e06a45798073b595ba982674" candidate_uuid: "3d49888c09ec4fb19d37486c92432902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f07a155a67c48ec8ca225d6a2438676"
I20260620 07:17:13.363410 19712 raft_consensus.cc:3037] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:13.364249 19712 raft_consensus.cc:2454] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d49888c09ec4fb19d37486c92432902 in term 1.
I20260620 07:17:13.364521 19572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092781a6e06a45798073b595ba982674" candidate_uuid: "3d49888c09ec4fb19d37486c92432902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485c95b97cdf45db8f4ca32c73bef46e"
I20260620 07:17:13.364564 19366 leader_election.cc:302] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d49888c09ec4fb19d37486c92432902, 4f07a155a67c48ec8ca225d6a2438676; no voters:
I20260620 07:17:13.364668 19572 raft_consensus.cc:3037] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:13.364728 19793 raft_consensus.cc:2781] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:13.365059 19793 raft_consensus.cc:686] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 1 LEADER]: Becoming Leader. State: Replica: 3d49888c09ec4fb19d37486c92432902, State: Running, Role: LEADER
I20260620 07:17:13.365342 19793 consensus_queue.cc:237] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.365893 19572 raft_consensus.cc:2454] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d49888c09ec4fb19d37486c92432902 in term 1.
I20260620 07:17:13.367733 19274 catalog_manager.cc:5495] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d49888c09ec4fb19d37486c92432902 (127.11.123.129). New cstate: current_term: 1 leader_uuid: "3d49888c09ec4fb19d37486c92432902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } health_report { overall_health: UNKNOWN } } }
W20260620 07:17:13.385023 19619 tablet.cc:2369] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:17:13 dist-test-slave-j663 krb5kdc[19230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939832, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.190@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:13.427810 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.418832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:42677 (local address 127.0.0.1:53412)
0620 07:17:13.419190 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.419204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.419318 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.419579 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.419582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.420058 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:13.420187 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.420194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.420896 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.420899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.421805 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.421815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.422294 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.422297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.422385 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.422979 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:13.422993 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:13.424563 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:13.426270 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:13.426280 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:13.426286 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:13.426513 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:13.426785 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:13.426790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:13.426792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:13.426875 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:13.427247 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:13.427255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:13.427355 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.427472 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.427621 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":106,"threads_started":1}
I20260620 07:17:13.427809 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.418914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.11.123.190:42677)
0620 07:17:13.419031 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.419036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.419294 (+ 258us) server_negotiation.cc:407] Connection header received
0620 07:17:13.419417 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.419419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.419461 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.419530 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:13.420297 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.420798 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.421970 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.422182 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.422271 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.424695 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:13.424699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:13.424701 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:13.424730 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:13.426146 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:13.426613 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:13.426619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:13.426621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:13.426696 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:13.426967 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:13.426971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:13.426972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:13.427138 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:13.427235 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.427525 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.427675 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":45}
I20260620 07:17:13.431697 19274 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:53412:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 07:17:13.432433 19274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260620 07:17:13.435343 19479 tablet.cc:2369] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:17:13.437244 19412 tablet_service.cc:1467] Processing CreateTablet for tablet d6f760bb766c4289a0487dae94ca6936 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=11abe946800c445db6949978052db3ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:17:13.437338 19552 tablet_service.cc:1467] Processing CreateTablet for tablet d6f760bb766c4289a0487dae94ca6936 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=11abe946800c445db6949978052db3ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:17:13.437630 19412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f760bb766c4289a0487dae94ca6936. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.437674 19552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f760bb766c4289a0487dae94ca6936. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.440448 19787 tablet_bootstrap.cc:492] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e: Bootstrap starting.
I20260620 07:17:13.441028 19692 tablet_service.cc:1467] Processing CreateTablet for tablet d6f760bb766c4289a0487dae94ca6936 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=11abe946800c445db6949978052db3ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:17:13.441424 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f760bb766c4289a0487dae94ca6936. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.441426 19787 tablet_bootstrap.cc:654] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.442081 19788 tablet_bootstrap.cc:492] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: Bootstrap starting.
I20260620 07:17:13.442571 19787 tablet_bootstrap.cc:492] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e: No bootstrap required, opened a new log
I20260620 07:17:13.442672 19787 ts_tablet_manager.cc:1397] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:13.443107 19788 tablet_bootstrap.cc:654] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.443071 19787 raft_consensus.cc:348] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.443178 19787 raft_consensus.cc:374] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.443218 19787 raft_consensus.cc:729] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 485c95b97cdf45db8f4ca32c73bef46e, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.443354 19787 consensus_queue.cc:260] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.443624 19787 ts_tablet_manager.cc:1428] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:17:13.444248 19806 tablet_replica.cc:439] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } } }
I20260620 07:17:13.444362 19786 tablet_bootstrap.cc:492] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676: Bootstrap starting.
I20260620 07:17:13.444396 19806 tablet_replica.cc:442] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:17:13.444522 19788 tablet_bootstrap.cc:492] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: No bootstrap required, opened a new log
I20260620 07:17:13.444635 19788 ts_tablet_manager.cc:1397] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 07:17:13.445113 19788 raft_consensus.cc:348] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.445231 19788 raft_consensus.cc:374] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.445279 19788 raft_consensus.cc:729] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d49888c09ec4fb19d37486c92432902, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.445354 19786 tablet_bootstrap.cc:654] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.445418 19788 consensus_queue.cc:260] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.445703 19793 tablet_replica.cc:439] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } } }
I20260620 07:17:13.445812 19793 tablet_replica.cc:442] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:17:13.446187 19788 ts_tablet_manager.cc:1428] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:17:13.446637 19786 tablet_bootstrap.cc:492] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676: No bootstrap required, opened a new log
I20260620 07:17:13.446730 19786 ts_tablet_manager.cc:1397] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 07:17:13.447222 19786 raft_consensus.cc:348] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.447340 19786 raft_consensus.cc:374] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.447376 19786 raft_consensus.cc:729] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f07a155a67c48ec8ca225d6a2438676, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.447487 19786 consensus_queue.cc:260] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.447772 19786 ts_tablet_manager.cc:1428] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:17:13.447724 19792 tablet_replica.cc:439] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } } }
I20260620 07:17:13.447865 19792 tablet_replica.cc:442] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:17:13.583352 19793 raft_consensus.cc:482] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:17:13.583536 19793 raft_consensus.cc:504] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.584003 19793 leader_election.cc:288] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130:43395), 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539)
I20260620 07:17:13.584403 19572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f760bb766c4289a0487dae94ca6936" candidate_uuid: "3d49888c09ec4fb19d37486c92432902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485c95b97cdf45db8f4ca32c73bef46e" is_pre_election: true
I20260620 07:17:13.584429 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f760bb766c4289a0487dae94ca6936" candidate_uuid: "3d49888c09ec4fb19d37486c92432902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f07a155a67c48ec8ca225d6a2438676" is_pre_election: true
I20260620 07:17:13.584551 19572 raft_consensus.cc:2454] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d49888c09ec4fb19d37486c92432902 in term 0.
I20260620 07:17:13.584599 19712 raft_consensus.cc:2454] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d49888c09ec4fb19d37486c92432902 in term 0.
I20260620 07:17:13.584898 19366 leader_election.cc:302] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d49888c09ec4fb19d37486c92432902, 4f07a155a67c48ec8ca225d6a2438676; no voters:
I20260620 07:17:13.585119 19793 raft_consensus.cc:2781] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:17:13.585184 19793 raft_consensus.cc:482] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:17:13.585213 19793 raft_consensus.cc:3037] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:13.586212 19793 raft_consensus.cc:504] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.586547 19793 leader_election.cc:288] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [CANDIDATE]: Term 1 election: Requested vote from peers 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130:43395), 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539)
I20260620 07:17:13.586872 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f760bb766c4289a0487dae94ca6936" candidate_uuid: "3d49888c09ec4fb19d37486c92432902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f07a155a67c48ec8ca225d6a2438676"
I20260620 07:17:13.587018 19712 raft_consensus.cc:3037] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:13.587049 19572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f760bb766c4289a0487dae94ca6936" candidate_uuid: "3d49888c09ec4fb19d37486c92432902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485c95b97cdf45db8f4ca32c73bef46e"
I20260620 07:17:13.587227 19572 raft_consensus.cc:3037] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:13.588063 19712 raft_consensus.cc:2454] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d49888c09ec4fb19d37486c92432902 in term 1.
I20260620 07:17:13.588336 19366 leader_election.cc:302] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d49888c09ec4fb19d37486c92432902, 4f07a155a67c48ec8ca225d6a2438676; no voters:
I20260620 07:17:13.588485 19572 raft_consensus.cc:2454] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d49888c09ec4fb19d37486c92432902 in term 1.
I20260620 07:17:13.588572 19793 raft_consensus.cc:2781] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:13.588668 19793 raft_consensus.cc:686] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [term 1 LEADER]: Becoming Leader. State: Replica: 3d49888c09ec4fb19d37486c92432902, State: Running, Role: LEADER
I20260620 07:17:13.588815 19793 consensus_queue.cc:237] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.589517 19817 tablet_replica.cc:439] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: TxnStatusTablet state changed. Reason: New leader 3d49888c09ec4fb19d37486c92432902. Latest consensus state: current_term: 1 leader_uuid: "3d49888c09ec4fb19d37486c92432902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } } }
I20260620 07:17:13.589634 19817 tablet_replica.cc:442] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:17:13.589931 19829 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 07:17:13.590180 19829 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 07:17:13.591017 19274 catalog_manager.cc:5495] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d49888c09ec4fb19d37486c92432902 (127.11.123.129). New cstate: current_term: 1 leader_uuid: "3d49888c09ec4fb19d37486c92432902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } health_report { overall_health: UNKNOWN } } }
I20260620 07:17:13.636267 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.632718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.11.123.129:45719)
0620 07:17:13.632836 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.632841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.632856 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:17:13.633025 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.633027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.633100 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.633207 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:17:13.633457 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.633956 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.634835 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.635056 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.635165 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.635461 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:17:13.635882 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:17:13.636017 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.636079 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.636126 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 07:17:13.636267 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.632577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45719 (local address 127.0.0.1:50206)
0620 07:17:13.632730 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.632745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.632850 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.633222 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.633225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.633233 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:17:13.633344 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.633355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.634077 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.634081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.634696 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.634704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.635167 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.635171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.635263 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.635309 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:17:13.635983 (+ 674us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:17:13.635999 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.636070 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.636154 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 07:17:13.639266 19712 raft_consensus.cc:1261] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Refusing update from remote peer 3d49888c09ec4fb19d37486c92432902: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:13.639731 19793 consensus_queue.cc:1035] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:13.639822 19572 raft_consensus.cc:1261] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Refusing update from remote peer 3d49888c09ec4fb19d37486c92432902: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:13.640364 19793 consensus_queue.cc:1035] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:13.642241 19806 tablet_replica.cc:439] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e: TxnStatusTablet state changed. Reason: New leader 3d49888c09ec4fb19d37486c92432902. Latest consensus state: current_term: 1 leader_uuid: "3d49888c09ec4fb19d37486c92432902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } } }
I20260620 07:17:13.642350 19806 tablet_replica.cc:442] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:17:13.642772 19793 tablet_replica.cc:439] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d49888c09ec4fb19d37486c92432902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } } }
I20260620 07:17:13.642904 19793 tablet_replica.cc:442] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:17:13.644374 19792 tablet_replica.cc:439] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676: TxnStatusTablet state changed. Reason: New leader 3d49888c09ec4fb19d37486c92432902. Latest consensus state: current_term: 1 leader_uuid: "3d49888c09ec4fb19d37486c92432902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } } }
I20260620 07:17:13.644487 19792 tablet_replica.cc:442] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:17:13.644580 19792 tablet_replica.cc:439] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d49888c09ec4fb19d37486c92432902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } } }
I20260620 07:17:13.644634 19801 mvcc.cc:205] Tried to move back new op lower bound from 7298825558580846592 to 7298825558381199360. Current Snapshot: MvccSnapshot[applied={T|T < 7298825558580846592 or (T in {7298825558580846592})}]
I20260620 07:17:13.644636 19792 tablet_replica.cc:442] T d6f760bb766c4289a0487dae94ca6936 P 4f07a155a67c48ec8ca225d6a2438676: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:17:13.646746 19817 tablet_replica.cc:439] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d49888c09ec4fb19d37486c92432902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } } }
I20260620 07:17:13.646852 19817 tablet_replica.cc:442] T d6f760bb766c4289a0487dae94ca6936 P 3d49888c09ec4fb19d37486c92432902: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:17:13.648223 19806 tablet_replica.cc:439] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d49888c09ec4fb19d37486c92432902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } } }
I20260620 07:17:13.648322 19806 tablet_replica.cc:442] T d6f760bb766c4289a0487dae94ca6936 P 485c95b97cdf45db8f4ca32c73bef46e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:17:13.652810 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.649643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.11.123.129:45719)
0620 07:17:13.649760 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.649766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.649781 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:17:13.649820 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.649822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.649864 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.649964 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:17:13.650223 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.650984 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.651771 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.652108 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.652210 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.652415 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:17:13.652495 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:17:13.652566 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.652642 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.652690 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 07:17:13.663085 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.659434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45719 (local address 127.11.123.129:37207)
0620 07:17:13.659739 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.659752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.659857 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.660081 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.660083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.660098 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:17:13.660209 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.660213 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.661188 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.661191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.661847 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.661857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.662418 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.662421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.662506 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.662543 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:17:13.662814 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:17:13.662819 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.662866 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.662922 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":16,"thread_start_us":91,"threads_started":1}
I20260620 07:17:13.663086 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.659749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:37207 (local address 127.11.123.129:45719)
0620 07:17:13.659876 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.659880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.659918 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:17:13.659955 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.659958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.660000 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.660066 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:17:13.660307 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.661054 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.661969 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.662297 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.662395 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.662667 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:17:13.662735 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:17:13.662806 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.662877 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.662921 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":53}
I20260620 07:17:13.669258 19712 raft_consensus.cc:1261] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Refusing update from remote peer 3d49888c09ec4fb19d37486c92432902: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:13.669554 19572 raft_consensus.cc:1261] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Refusing update from remote peer 3d49888c09ec4fb19d37486c92432902: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:13.669584 19838 consensus_queue.cc:1035] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:13.669862 19838 consensus_queue.cc:1035] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:13.706102 19274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:13.706262 19274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:13.708140 19274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005: Sending DeleteTablet for 3 replicas of tablet 092781a6e06a45798073b595ba982674
I20260620 07:17:13.708580 19411 tablet_service.cc:1514] Processing DeleteTablet for tablet 092781a6e06a45798073b595ba982674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:13 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:48986
I20260620 07:17:13.708727 19810 tablet_replica.cc:330] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902: stopping tablet replica
I20260620 07:17:13.708819 19810 raft_consensus.cc:2229] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:17:13.708982 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet 092781a6e06a45798073b595ba982674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:13 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:41472
I20260620 07:17:13.709065 19810 raft_consensus.cc:2258] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:13.709208 19811 tablet_replica.cc:330] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676: stopping tablet replica
I20260620 07:17:13.709309 19811 raft_consensus.cc:2229] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:13.709407 19811 raft_consensus.cc:2258] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:13.709794 19810 ts_tablet_manager.cc:1905] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:13.709931 19552 tablet_service.cc:1514] Processing DeleteTablet for tablet 092781a6e06a45798073b595ba982674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:13 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:34808
I20260620 07:17:13.710445 19809 tablet_replica.cc:330] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e: stopping tablet replica
I20260620 07:17:13.710543 19809 raft_consensus.cc:2229] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:13.710637 19809 raft_consensus.cc:2258] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:13.712483 19810 ts_tablet_manager.cc:1918] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:17:13.712553 19810 log.cc:1196] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/wal/wals/092781a6e06a45798073b595ba982674
I20260620 07:17:13.712786 19810 ts_tablet_manager.cc:1939] T 092781a6e06a45798073b595ba982674 P 3d49888c09ec4fb19d37486c92432902: Deleting consensus metadata
I20260620 07:17:13.714470 19809 ts_tablet_manager.cc:1905] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:13.716985 19809 ts_tablet_manager.cc:1918] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:17:13.717057 19809 log.cc:1196] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/092781a6e06a45798073b595ba982674
I20260620 07:17:13.717485 19809 ts_tablet_manager.cc:1939] T 092781a6e06a45798073b595ba982674 P 485c95b97cdf45db8f4ca32c73bef46e: Deleting consensus metadata
I20260620 07:17:13.718194 19259 catalog_manager.cc:4841] TS 3d49888c09ec4fb19d37486c92432902 (127.11.123.129:45719): tablet 092781a6e06a45798073b595ba982674 (table test-table [id=884a0df63ab64c23ab5d105025c1c95b]) successfully deleted
I20260620 07:17:13.718217 19261 catalog_manager.cc:4841] TS 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130:43395): tablet 092781a6e06a45798073b595ba982674 (table test-table [id=884a0df63ab64c23ab5d105025c1c95b]) successfully deleted
I20260620 07:17:13.718977 19811 ts_tablet_manager.cc:1905] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:13.721762 19811 ts_tablet_manager.cc:1918] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:17:13.721848 19811 log.cc:1196] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/092781a6e06a45798073b595ba982674
I20260620 07:17:13.722110 19811 ts_tablet_manager.cc:1939] T 092781a6e06a45798073b595ba982674 P 4f07a155a67c48ec8ca225d6a2438676: Deleting consensus metadata
I20260620 07:17:13.722792 19262 catalog_manager.cc:4841] TS 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539): tablet 092781a6e06a45798073b595ba982674 (table test-table [id=884a0df63ab64c23ab5d105025c1c95b]) successfully deleted
Jun 20 07:17:13 dist-test-slave-j663 krb5kdc[19230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:17:13 dist-test-slave-j663 krb5kdc[19230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:13 dist-test-slave-j663 krb5kdc[19230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:17:13.743804 19324 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:13.736727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.11.123.190:42677)
0620 07:17:13.736878 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.736883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.737012 (+ 129us) server_negotiation.cc:407] Connection header received
0620 07:17:13.737213 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.737217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.737276 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.737372 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:13.738073 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.738862 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.739702 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.740062 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.740178 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.743684 (+ 3506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53416: BlockingRecv error: recv got EOF from 127.0.0.1:53416 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 07:17:13 dist-test-slave-j663 krb5kdc[19230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM
I20260620 07:17:13.753592 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.745145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53420 (local address 127.11.123.190:42677)
0620 07:17:13.745274 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.745279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.745575 (+ 296us) server_negotiation.cc:407] Connection header received
0620 07:17:13.745643 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.745646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.745693 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.745772 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:13.746333 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.747089 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.747827 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.748153 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.748253 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.750560 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:13.750564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:13.750567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:13.750591 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:13.752051 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:13.752419 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:13.752422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:13.752424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:13.752506 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:13.752738 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:13.752741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:13.752743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:13.752936 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:13.753094 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.753338 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.753440 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:17:13.756201 19274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:17:13.756649 19274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:17:13.761495 19411 tablet_service.cc:1467] Processing CreateTablet for tablet 8a12896219dd49df9f607dd7a3be081c (DEFAULT_TABLE table=test-table [id=fc639a7a37a84057acd0dd369b6db6b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:13.761691 19552 tablet_service.cc:1467] Processing CreateTablet for tablet 8a12896219dd49df9f607dd7a3be081c (DEFAULT_TABLE table=test-table [id=fc639a7a37a84057acd0dd369b6db6b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:13.761833 19411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a12896219dd49df9f607dd7a3be081c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.762006 19552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a12896219dd49df9f607dd7a3be081c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.763816 19788 tablet_bootstrap.cc:492] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902: Bootstrap starting.
I20260620 07:17:13.764209 19692 tablet_service.cc:1467] Processing CreateTablet for tablet 8a12896219dd49df9f607dd7a3be081c (DEFAULT_TABLE table=test-table [id=fc639a7a37a84057acd0dd369b6db6b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:13.764478 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a12896219dd49df9f607dd7a3be081c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.764588 19787 tablet_bootstrap.cc:492] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e: Bootstrap starting.
I20260620 07:17:13.764652 19788 tablet_bootstrap.cc:654] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.765756 19787 tablet_bootstrap.cc:654] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.766582 19786 tablet_bootstrap.cc:492] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676: Bootstrap starting.
I20260620 07:17:13.766835 19787 tablet_bootstrap.cc:492] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e: No bootstrap required, opened a new log
I20260620 07:17:13.766901 19787 ts_tablet_manager.cc:1397] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:13.767292 19787 raft_consensus.cc:348] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.767390 19787 raft_consensus.cc:374] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.767418 19787 raft_consensus.cc:729] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 485c95b97cdf45db8f4ca32c73bef46e, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.767527 19786 tablet_bootstrap.cc:654] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.767526 19787 consensus_queue.cc:260] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.767601 19788 tablet_bootstrap.cc:492] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902: No bootstrap required, opened a new log
I20260620 07:17:13.767679 19788 ts_tablet_manager.cc:1397] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 07:17:13.767746 19787 ts_tablet_manager.cc:1428] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:17:13.768110 19788 raft_consensus.cc:348] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.768229 19788 raft_consensus.cc:374] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.768270 19788 raft_consensus.cc:729] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d49888c09ec4fb19d37486c92432902, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.768415 19788 consensus_queue.cc:260] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.768754 19788 ts_tablet_manager.cc:1428] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 07:17:13.769068 19786 tablet_bootstrap.cc:492] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676: No bootstrap required, opened a new log
I20260620 07:17:13.769183 19786 ts_tablet_manager.cc:1397] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 07:17:13.769867 19786 raft_consensus.cc:348] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.769997 19786 raft_consensus.cc:374] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.770031 19786 raft_consensus.cc:729] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f07a155a67c48ec8ca225d6a2438676, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.770154 19786 consensus_queue.cc:260] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.770429 19786 ts_tablet_manager.cc:1428] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:17:13.836350 19834 raft_consensus.cc:482] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:17:13.836510 19834 raft_consensus.cc:504] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.837698 19834 leader_election.cc:288] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d49888c09ec4fb19d37486c92432902 (127.11.123.129:45719), 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130:43395)
I20260620 07:17:13.842356 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.838386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:57101 (local address 127.11.123.130:43395)
0620 07:17:13.838522 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.838526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.838537 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:17:13.838580 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.838582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.838619 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.838720 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:13.839009 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.839547 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.841125 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.841762 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.841896 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.841947 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.842155 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.842227 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:17:13.842382 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.837918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:43395 (local address 127.11.123.131:57101)
0620 07:17:13.838280 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.838296 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.838399 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.838741 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.838744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.838760 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:13.838879 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.838885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.839665 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.839672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.840989 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.840999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.841916 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.841920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.842026 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.842053 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.842126 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.842207 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":60,"thread_start_us":100,"threads_started":1}
I20260620 07:17:13.842804 19572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a12896219dd49df9f607dd7a3be081c" candidate_uuid: "4f07a155a67c48ec8ca225d6a2438676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485c95b97cdf45db8f4ca32c73bef46e" is_pre_election: true
I20260620 07:17:13.842979 19572 raft_consensus.cc:2454] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f07a155a67c48ec8ca225d6a2438676 in term 0.
I20260620 07:17:13.843008 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.838164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:34155 (local address 127.11.123.129:45719)
0620 07:17:13.838282 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.838288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.838514 (+ 226us) server_negotiation.cc:407] Connection header received
0620 07:17:13.838708 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.838713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.838759 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.838830 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:13.839147 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.839902 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.841349 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.841953 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.842075 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.842141 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.842815 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.842892 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:17:13.843271 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.837920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45719 (local address 127.11.123.131:34155)
0620 07:17:13.838411 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:13.838436 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:13.838582 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:13.838854 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:13.838857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:13.838875 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:13.839020 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.839027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.840043 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.840047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.841233 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:13.841240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.842215 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.842218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:13.842321 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.842343 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:17:13.842778 (+ 435us) client_negotiation.cc:240] Negotiation successful
0620 07:17:13.842855 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":209,"thread_start_us":58,"threads_started":1}
I20260620 07:17:13.843387 19645 leader_election.cc:302] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 485c95b97cdf45db8f4ca32c73bef46e, 4f07a155a67c48ec8ca225d6a2438676; no voters:
I20260620 07:17:13.843694 19834 raft_consensus.cc:2781] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:17:13.843761 19834 raft_consensus.cc:482] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:17:13.843799 19834 raft_consensus.cc:3037] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:13.843815 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a12896219dd49df9f607dd7a3be081c" candidate_uuid: "4f07a155a67c48ec8ca225d6a2438676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d49888c09ec4fb19d37486c92432902" is_pre_election: true
I20260620 07:17:13.843959 19432 raft_consensus.cc:2454] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f07a155a67c48ec8ca225d6a2438676 in term 0.
I20260620 07:17:13.845036 19834 raft_consensus.cc:504] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.845398 19834 leader_election.cc:288] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [CANDIDATE]: Term 1 election: Requested vote from peers 3d49888c09ec4fb19d37486c92432902 (127.11.123.129:45719), 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130:43395)
I20260620 07:17:13.845739 19572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a12896219dd49df9f607dd7a3be081c" candidate_uuid: "4f07a155a67c48ec8ca225d6a2438676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485c95b97cdf45db8f4ca32c73bef46e"
I20260620 07:17:13.845759 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a12896219dd49df9f607dd7a3be081c" candidate_uuid: "4f07a155a67c48ec8ca225d6a2438676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d49888c09ec4fb19d37486c92432902"
I20260620 07:17:13.845851 19572 raft_consensus.cc:3037] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:13.845889 19432 raft_consensus.cc:3037] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:13.846781 19432 raft_consensus.cc:2454] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f07a155a67c48ec8ca225d6a2438676 in term 1.
I20260620 07:17:13.846940 19572 raft_consensus.cc:2454] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f07a155a67c48ec8ca225d6a2438676 in term 1.
I20260620 07:17:13.847105 19643 leader_election.cc:302] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d49888c09ec4fb19d37486c92432902, 4f07a155a67c48ec8ca225d6a2438676; no voters:
I20260620 07:17:13.847258 19834 raft_consensus.cc:2781] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:13.847359 19834 raft_consensus.cc:686] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 1 LEADER]: Becoming Leader. State: Replica: 4f07a155a67c48ec8ca225d6a2438676, State: Running, Role: LEADER
I20260620 07:17:13.847594 19834 consensus_queue.cc:237] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.849922 19274 catalog_manager.cc:5495] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131). New cstate: current_term: 1 leader_uuid: "4f07a155a67c48ec8ca225d6a2438676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } health_report { overall_health: HEALTHY } } }
I20260620 07:17:13.870910 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:13.867167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.11.123.131:37539)
0620 07:17:13.867414 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:13.867418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:13.867430 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:17:13.867498 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:13.867500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:13.867546 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:13.867710 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:17:13.867955 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.868449 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.869376 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:13.869608 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:13.869719 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:13.869996 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:17:13.870525 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:17:13.870655 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:13.870714 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 07:17:13.870766 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
I20260620 07:17:13.873534 19432 raft_consensus.cc:1261] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Refusing update from remote peer 4f07a155a67c48ec8ca225d6a2438676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:13.873541 19572 raft_consensus.cc:1261] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Refusing update from remote peer 4f07a155a67c48ec8ca225d6a2438676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:13.874009 19834 consensus_queue.cc:1035] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:13.874586 19834 consensus_queue.cc:1035] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:13.889763 19275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:13.889938 19275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:13.892270 19275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005: Sending DeleteTablet for 3 replicas of tablet 8a12896219dd49df9f607dd7a3be081c
I20260620 07:17:13.892771 19411 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a12896219dd49df9f607dd7a3be081c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:13 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:48986
I20260620 07:17:13.892957 19810 tablet_replica.cc:330] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902: stopping tablet replica
I20260620 07:17:13.893112 19810 raft_consensus.cc:2229] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:13.893250 19810 raft_consensus.cc:2258] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:13.894025 19552 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a12896219dd49df9f607dd7a3be081c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:13 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:34808
I20260620 07:17:13.894228 19809 tablet_replica.cc:330] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e: stopping tablet replica
I20260620 07:17:13.894352 19809 raft_consensus.cc:2229] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:13.894472 19809 raft_consensus.cc:2258] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:13.894922 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a12896219dd49df9f607dd7a3be081c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:13 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:41472
I20260620 07:17:13.895004 19810 ts_tablet_manager.cc:1905] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:13.895079 19811 tablet_replica.cc:330] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676: stopping tablet replica
I20260620 07:17:13.895172 19811 raft_consensus.cc:2229] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:17:13.895453 19811 raft_consensus.cc:2258] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:13.896258 19809 ts_tablet_manager.cc:1905] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:13.897523 19810 ts_tablet_manager.cc:1918] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:13.897614 19810 log.cc:1196] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-0/wal/wals/8a12896219dd49df9f607dd7a3be081c
I20260620 07:17:13.897715 19811 ts_tablet_manager.cc:1905] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:13.897872 19810 ts_tablet_manager.cc:1939] T 8a12896219dd49df9f607dd7a3be081c P 3d49888c09ec4fb19d37486c92432902: Deleting consensus metadata
I20260620 07:17:13.898511 19259 catalog_manager.cc:4841] TS 3d49888c09ec4fb19d37486c92432902 (127.11.123.129:45719): tablet 8a12896219dd49df9f607dd7a3be081c (table test-table [id=fc639a7a37a84057acd0dd369b6db6b6]) successfully deleted
I20260620 07:17:13.900137 19811 ts_tablet_manager.cc:1918] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:13.900205 19811 log.cc:1196] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/8a12896219dd49df9f607dd7a3be081c
I20260620 07:17:13.900476 19811 ts_tablet_manager.cc:1939] T 8a12896219dd49df9f607dd7a3be081c P 4f07a155a67c48ec8ca225d6a2438676: Deleting consensus metadata
I20260620 07:17:13.901171 19262 catalog_manager.cc:4841] TS 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539): tablet 8a12896219dd49df9f607dd7a3be081c (table test-table [id=fc639a7a37a84057acd0dd369b6db6b6]) successfully deleted
I20260620 07:17:13.901309 19274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:17:13.901744 19274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:17:13.904431 19809 ts_tablet_manager.cc:1918] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:13.908355 19692 tablet_service.cc:1467] Processing CreateTablet for tablet ad0bdba84f914995aef8aac8c7908012 (DEFAULT_TABLE table=test-table [id=bc5e1b439b224c31b4ef9b299e787767]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:13.908587 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0bdba84f914995aef8aac8c7908012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.911306 19411 tablet_service.cc:1467] Processing CreateTablet for tablet ad0bdba84f914995aef8aac8c7908012 (DEFAULT_TABLE table=test-table [id=bc5e1b439b224c31b4ef9b299e787767]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:13.911556 19411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0bdba84f914995aef8aac8c7908012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.913897 19788 tablet_bootstrap.cc:492] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902: Bootstrap starting.
I20260620 07:17:13.914690 19788 tablet_bootstrap.cc:654] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.915818 19786 tablet_bootstrap.cc:492] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676: Bootstrap starting.
I20260620 07:17:13.916671 19786 tablet_bootstrap.cc:654] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.917909 19809 log.cc:1196] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/8a12896219dd49df9f607dd7a3be081c
I20260620 07:17:13.910691 19552 tablet_service.cc:1467] Processing CreateTablet for tablet ad0bdba84f914995aef8aac8c7908012 (DEFAULT_TABLE table=test-table [id=bc5e1b439b224c31b4ef9b299e787767]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:13.918176 19552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0bdba84f914995aef8aac8c7908012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:13.920517 19787 tablet_bootstrap.cc:492] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e: Bootstrap starting.
I20260620 07:17:13.921286 19787 tablet_bootstrap.cc:654] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:13.923147 19809 ts_tablet_manager.cc:1939] T 8a12896219dd49df9f607dd7a3be081c P 485c95b97cdf45db8f4ca32c73bef46e: Deleting consensus metadata
I20260620 07:17:13.923861 19261 catalog_manager.cc:4841] TS 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130:43395): tablet 8a12896219dd49df9f607dd7a3be081c (table test-table [id=fc639a7a37a84057acd0dd369b6db6b6]) successfully deleted
I20260620 07:17:13.924226 19787 tablet_bootstrap.cc:492] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e: No bootstrap required, opened a new log
I20260620 07:17:13.924317 19787 ts_tablet_manager.cc:1397] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 07:17:13.924695 19787 raft_consensus.cc:348] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.924831 19787 raft_consensus.cc:374] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.924865 19787 raft_consensus.cc:729] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 485c95b97cdf45db8f4ca32c73bef46e, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.925002 19787 consensus_queue.cc:260] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.925448 19787 ts_tablet_manager.cc:1428] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 07:17:13.926159 19788 tablet_bootstrap.cc:492] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902: No bootstrap required, opened a new log
I20260620 07:17:13.926234 19788 ts_tablet_manager.cc:1397] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20260620 07:17:13.926502 19786 tablet_bootstrap.cc:492] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676: No bootstrap required, opened a new log
I20260620 07:17:13.926580 19786 ts_tablet_manager.cc:1397] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20260620 07:17:13.926585 19788 raft_consensus.cc:348] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.926683 19788 raft_consensus.cc:374] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.926710 19788 raft_consensus.cc:729] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d49888c09ec4fb19d37486c92432902, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.926815 19788 consensus_queue.cc:260] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.927030 19788 ts_tablet_manager.cc:1428] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:17:13.926988 19786 raft_consensus.cc:348] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.927089 19786 raft_consensus.cc:374] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:13.927120 19786 raft_consensus.cc:729] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f07a155a67c48ec8ca225d6a2438676, State: Initialized, Role: FOLLOWER
I20260620 07:17:13.927253 19786 consensus_queue.cc:260] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:13.927541 19786 ts_tablet_manager.cc:1428] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 07:17:14.026868 19806 raft_consensus.cc:482] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:17:14.027025 19806 raft_consensus.cc:504] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:14.027516 19806 leader_election.cc:288] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d49888c09ec4fb19d37486c92432902 (127.11.123.129:45719), 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539)
I20260620 07:17:14.027930 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0bdba84f914995aef8aac8c7908012" candidate_uuid: "485c95b97cdf45db8f4ca32c73bef46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f07a155a67c48ec8ca225d6a2438676" is_pre_election: true
I20260620 07:17:14.027956 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0bdba84f914995aef8aac8c7908012" candidate_uuid: "485c95b97cdf45db8f4ca32c73bef46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d49888c09ec4fb19d37486c92432902" is_pre_election: true
I20260620 07:17:14.028085 19712 raft_consensus.cc:2454] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 485c95b97cdf45db8f4ca32c73bef46e in term 0.
I20260620 07:17:14.028090 19432 raft_consensus.cc:2454] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 485c95b97cdf45db8f4ca32c73bef46e in term 0.
I20260620 07:17:14.028353 19506 leader_election.cc:302] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 485c95b97cdf45db8f4ca32c73bef46e, 4f07a155a67c48ec8ca225d6a2438676; no voters:
I20260620 07:17:14.028540 19806 raft_consensus.cc:2781] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:17:14.028605 19806 raft_consensus.cc:482] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:17:14.028662 19806 raft_consensus.cc:3037] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:14.029688 19806 raft_consensus.cc:504] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:14.030040 19806 leader_election.cc:288] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [CANDIDATE]: Term 1 election: Requested vote from peers 3d49888c09ec4fb19d37486c92432902 (127.11.123.129:45719), 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539)
I20260620 07:17:14.030357 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0bdba84f914995aef8aac8c7908012" candidate_uuid: "485c95b97cdf45db8f4ca32c73bef46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f07a155a67c48ec8ca225d6a2438676"
I20260620 07:17:14.030385 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0bdba84f914995aef8aac8c7908012" candidate_uuid: "485c95b97cdf45db8f4ca32c73bef46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d49888c09ec4fb19d37486c92432902"
I20260620 07:17:14.030478 19712 raft_consensus.cc:3037] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:14.030498 19432 raft_consensus.cc:3037] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:14.031560 19712 raft_consensus.cc:2454] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485c95b97cdf45db8f4ca32c73bef46e in term 1.
I20260620 07:17:14.031560 19432 raft_consensus.cc:2454] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485c95b97cdf45db8f4ca32c73bef46e in term 1.
I20260620 07:17:14.031817 19506 leader_election.cc:302] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 485c95b97cdf45db8f4ca32c73bef46e, 4f07a155a67c48ec8ca225d6a2438676; no voters:
I20260620 07:17:14.032028 19806 raft_consensus.cc:2781] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:14.032246 19806 raft_consensus.cc:686] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 LEADER]: Becoming Leader. State: Replica: 485c95b97cdf45db8f4ca32c73bef46e, State: Running, Role: LEADER
I20260620 07:17:14.032418 19806 consensus_queue.cc:237] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } }
I20260620 07:17:14.034087 19275 catalog_manager.cc:5495] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e reported cstate change: term changed from 0 to 1, leader changed from <none> to 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130). New cstate: current_term: 1 leader_uuid: "485c95b97cdf45db8f4ca32c73bef46e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485c95b97cdf45db8f4ca32c73bef46e" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 43395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 } health_report { overall_health: UNKNOWN } } }
I20260620 07:17:14.056643 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:14.053592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.11.123.130:43395)
0620 07:17:14.053705 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:14.053709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:14.053763 (+ 54us) server_negotiation.cc:407] Connection header received
0620 07:17:14.053870 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:14.053872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:14.053916 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:14.053991 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:17:14.054242 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:14.055041 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:14.055802 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:14.056046 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:14.056128 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:14.056317 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:17:14.056377 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:17:14.056429 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:14.056503 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 07:17:14.056543 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:17:14.065409 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:14.062639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:59461 (local address 127.11.123.129:45719)
0620 07:17:14.062762 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:14.062766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:14.062938 (+ 172us) server_negotiation.cc:407] Connection header received
0620 07:17:14.063053 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:14.063055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:14.063094 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:14.063154 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:17:14.063345 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:14.063791 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:14.064537 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:14.064731 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:14.064801 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:14.065021 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:17:14.065118 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:17:14.065184 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:14.065281 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 07:17:14.065318 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:17:14.065421 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:14.062591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45719 (local address 127.11.123.130:59461)
0620 07:17:14.062866 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:14.062878 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:14.062958 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:14.063180 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:14.063182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:14.063187 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:17:14.063260 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:14.063264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:14.063873 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:14.063875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:14.064445 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:14.064452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:14.064824 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:14.064827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:14.064883 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:14.064918 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:17:14.065204 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:17:14.065209 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:17:14.065257 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 07:17:14.065315 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":114,"threads_started":1}
I20260620 07:17:14.070161 19712 raft_consensus.cc:1261] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Refusing update from remote peer 485c95b97cdf45db8f4ca32c73bef46e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:14.070194 19432 raft_consensus.cc:1261] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Refusing update from remote peer 485c95b97cdf45db8f4ca32c73bef46e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:14.070504 19868 consensus_queue.cc:1035] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f07a155a67c48ec8ca225d6a2438676" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:14.070674 19806 consensus_queue.cc:1035] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d49888c09ec4fb19d37486c92432902" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:14.085600 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:14.082523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:32817 (local address 127.11.123.130:43395)
0620 07:17:14.082640 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:14.082645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:14.082707 (+ 62us) server_negotiation.cc:407] Connection header received
0620 07:17:14.082819 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:14.082823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:14.082863 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:14.082933 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:17:14.083188 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:14.083859 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:14.084616 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:14.084808 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:14.084874 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:14.085065 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:17:14.085168 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:17:14.085259 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:14.085394 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 07:17:14.085448 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:17:14.085615 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:14.082470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:43395 (local address 127.11.123.129:32817)
0620 07:17:14.082646 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:14.082661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:14.082743 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:14.082959 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:14.082961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:14.082968 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:17:14.083044 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:14.083048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:14.083958 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:14.083962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:14.084509 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:14.084517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:14.084893 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:14.084895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:14.084951 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:14.084973 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:17:14.085302 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:17:14.085311 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:17:14.085375 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 07:17:14.085461 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 07:17:14.104230 19275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:14.104368 19275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:14.106266 19275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f02ab269a6048dab9eb3976b5f6f005: Sending DeleteTablet for 3 replicas of tablet ad0bdba84f914995aef8aac8c7908012
I20260620 07:17:14.106726 19552 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0bdba84f914995aef8aac8c7908012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:14 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:34808
I20260620 07:17:14.106822 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0bdba84f914995aef8aac8c7908012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:14 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:41472
I20260620 07:17:14.106916 19809 tablet_replica.cc:330] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e: stopping tablet replica
I20260620 07:17:14.106999 19811 tablet_replica.cc:330] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676: stopping tablet replica
I20260620 07:17:14.107057 19809 raft_consensus.cc:2229] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:17:14.107111 19811 raft_consensus.cc:2229] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:14.107162 19411 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0bdba84f914995aef8aac8c7908012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:14 UTC) from {username='oryx', principal='oryx/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:48986
I20260620 07:17:14.107218 19811 raft_consensus.cc:2258] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:14.107239 19809 raft_consensus.cc:2258] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:14.107362 19810 tablet_replica.cc:330] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902: stopping tablet replica
I20260620 07:17:14.107487 19810 raft_consensus.cc:2229] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:14.107614 19810 raft_consensus.cc:2258] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:14.108335 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 19347
I20260620 07:17:14.108369 19810 ts_tablet_manager.cc:1905] T ad0bdba84f914995aef8aac8c7908012 P 3d49888c09ec4fb19d37486c92432902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:14.110097 19811 ts_tablet_manager.cc:1905] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:14.111603 19809 ts_tablet_manager.cc:1905] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:14.112654 19811 ts_tablet_manager.cc:1918] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:17:14.112721 19811 log.cc:1196] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/ad0bdba84f914995aef8aac8c7908012
I20260620 07:17:14.113221 19811 ts_tablet_manager.cc:1939] T ad0bdba84f914995aef8aac8c7908012 P 4f07a155a67c48ec8ca225d6a2438676: Deleting consensus metadata
I20260620 07:17:14.113906 19262 catalog_manager.cc:4841] TS 4f07a155a67c48ec8ca225d6a2438676 (127.11.123.131:37539): tablet ad0bdba84f914995aef8aac8c7908012 (table test-table [id=bc5e1b439b224c31b4ef9b299e787767]) successfully deleted
I20260620 07:17:14.114070 19809 ts_tablet_manager.cc:1918] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:17:14.114130 19809 log.cc:1196] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/ad0bdba84f914995aef8aac8c7908012
I20260620 07:17:14.114319 19809 ts_tablet_manager.cc:1939] T ad0bdba84f914995aef8aac8c7908012 P 485c95b97cdf45db8f4ca32c73bef46e: Deleting consensus metadata
I20260620 07:17:14.114913 19261 catalog_manager.cc:4841] TS 485c95b97cdf45db8f4ca32c73bef46e (127.11.123.130:43395): tablet ad0bdba84f914995aef8aac8c7908012 (table test-table [id=bc5e1b439b224c31b4ef9b299e787767]) successfully deleted
W20260620 07:17:14.118580 19259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:45719 (error 108)
I20260620 07:17:14.119277 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 19487
I20260620 07:17:14.120333 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:14.119273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45719 (local address 127.0.0.1:50228)
0620 07:17:14.120173 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:14.120246 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:45719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":56,"threads_started":1}
W20260620 07:17:14.120435 19259 catalog_manager.cc:4584] TS 3d49888c09ec4fb19d37486c92432902 (127.11.123.129:45719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad0bdba84f914995aef8aac8c7908012: Network error: Client connection negotiation failed: client connection to 127.11.123.129:45719: connect: Connection refused (error 111)
I20260620 07:17:14.126305 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 19627
I20260620 07:17:14.131606 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 19243
2026-06-20T07:17:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3039 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:17:14 dist-test-slave-j663 krb5kdc[19881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:17:14 dist-test-slave-j663 krb5kdc[19881](info): set up 2 sockets
Jun 20 07:17:14 dist-test-slave-j663 krb5kdc[19881](info): commencing operation
krb5kdc: starting...
W20260620 07:17:15.173815 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:17:15 dist-test-slave-j663 krb5kdc[19881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:17:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:17:15.291679 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:46043
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:46043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:15.399374 19894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:15.399616 19894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:15.399655 19894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:15.403515 19894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:15.403566 19894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:15.403585 19894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:15.403601 19894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:15.403616 19894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:17:15.408676 19894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:46043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:46043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:15.409606 19894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:15.410518 19894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:15.418609 19899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:15.418756 19894 server_base.cc:1034] running on GCE node
W20260620 07:17:15.418629 19902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:15.418955 19900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:15.419477 19894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:15.420526 19894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:15.421710 19894 hybrid_clock.cc:648] HybridClock initialized: now 1781939835421662 us; error 60 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T07:17:15Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1296 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:17:15 dist-test-slave-j663 krb5kdc[19907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:17:15 dist-test-slave-j663 krb5kdc[19907](info): set up 2 sockets
Jun 20 07:17:15 dist-test-slave-j663 krb5kdc[19907](info): commencing operation
krb5kdc: starting...
W20260620 07:17:16.466010 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:17:16 dist-test-slave-j663 krb5kdc[19907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:17:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:17:16.587786 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:37605
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:37605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:16.694051 19920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:16.694286 19920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:16.694324 19920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:16.698231 19920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:16.698283 19920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:16.698302 19920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:16.698318 19920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:16.698379 19920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:17:16.703529 19920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:37605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:37605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:16.704478 19920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:16.705458 19920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:16.713420 19926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:16.713476 19925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:16.713420 19928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:16.713616 19920 server_base.cc:1034] running on GCE node
I20260620 07:17:16.714192 19920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:16.715243 19920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:16.716406 19920 hybrid_clock.cc:648] HybridClock initialized: now 1781939836716378 us; error 38 us; skew 500 ppm
Jun 20 07:17:16 dist-test-slave-j663 krb5kdc[19907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939836, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:16.719357 19920 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:17:16.720156 19920 webserver.cc:460] Webserver started at http://127.11.123.190:44737/ using document root <none> and password file <none>
I20260620 07:17:16.720655 19920 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:16.720712 19920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:16.720870 19920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:16.723119 19920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "b2f6f2b23e1e45a287268deb3b3c6fec"
format_stamp: "Formatted at 2026-06-20 07:17:16 on dist-test-slave-j663"
server_key: "8d619d75ab72ffd39cb31e3eda3bc2c2"
server_key_iv: "1ee4241876eea3cc62da408542c79879"
server_key_version: "encryptionkey@0"
I20260620 07:17:16.723623 19920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "b2f6f2b23e1e45a287268deb3b3c6fec"
format_stamp: "Formatted at 2026-06-20 07:17:16 on dist-test-slave-j663"
server_key: "8d619d75ab72ffd39cb31e3eda3bc2c2"
server_key_iv: "1ee4241876eea3cc62da408542c79879"
server_key_version: "encryptionkey@0"
I20260620 07:17:16.727196 19920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:17:16.729487 19935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:16.730671 19920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:16.730815 19920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "b2f6f2b23e1e45a287268deb3b3c6fec"
format_stamp: "Formatted at 2026-06-20 07:17:16 on dist-test-slave-j663"
server_key: "8d619d75ab72ffd39cb31e3eda3bc2c2"
server_key_iv: "1ee4241876eea3cc62da408542c79879"
server_key_version: "encryptionkey@0"
I20260620 07:17:16.730924 19920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:16.757475 19920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:16.760236 19920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:16.760437 19920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:16.767570 19920 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:37605
I20260620 07:17:16.767592 19987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:37605 every 8 connection(s)
I20260620 07:17:16.768467 19920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:16.771198 19988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:16.773715 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 19920
I20260620 07:17:16.773844 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:17:16.774154 11758 external_mini_cluster.cc:1428] Setting key a74bb75f8158d5f9b6993414f011e8e8
I20260620 07:17:16.777057 19988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec: Bootstrap starting.
I20260620 07:17:16.779508 19988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:16.780373 19988 log.cc:824] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:16.782306 19988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec: No bootstrap required, opened a new log
Jun 20 07:17:16 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:16.785552 19988 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37605 } }
I20260620 07:17:16.785784 19988 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:16.785872 19988 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f6f2b23e1e45a287268deb3b3c6fec, State: Initialized, Role: FOLLOWER
I20260620 07:17:16.786347 19988 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37605 } }
I20260620 07:17:16.786490 19988 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:16.786576 19988 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:16.786674 19988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:16.787500 19988 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37605 } }
I20260620 07:17:16.787834 19988 leader_election.cc:302] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2f6f2b23e1e45a287268deb3b3c6fec; no voters:
I20260620 07:17:16.788194 19988 leader_election.cc:288] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:17:16.788257 19993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:16.788466 19993 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 1 LEADER]: Becoming Leader. State: Replica: b2f6f2b23e1e45a287268deb3b3c6fec, State: Running, Role: LEADER
I20260620 07:17:16.788813 19993 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37605 } }
I20260620 07:17:16.789172 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:16.775434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38400 (local address 127.11.123.190:37605)
0620 07:17:16.775819 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:16.775834 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:16.775875 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:17:16.776471 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:16.776488 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:16.776802 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:16.777139 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:16.778029 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:16.779004 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:16.780187 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:16.780705 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:16.780889 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:16.783474 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:16.783480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:16.783493 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:16.783520 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:16.785839 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:16.786354 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:16.786359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:16.786364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:16.786505 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:16.786861 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:16.786867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:16.786869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:16.787267 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:16.787501 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:16.787776 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 07:17:16.788106 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":128,"threads_started":1}
I20260620 07:17:16.789443 19988 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:16.789958 19993 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f6f2b23e1e45a287268deb3b3c6fec. Latest consensus state: current_term: 1 leader_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37605 } } }
I20260620 07:17:16.791779 19993 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:16.791797 19994 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37605 } } }
I20260620 07:17:16.792410 19994 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [sys.catalog]: This master's current role is: LEADER
W20260620 07:17:16.794263 20007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:17:16.794330 20007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:17:16.794415 20009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:16.795194 20009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:16.800973 20009 catalog_manager.cc:1347] Generated new cluster ID: 1113fa9c90a64e71b6f0c9b2c96d6a33
I20260620 07:17:16.801038 20009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:16.832068 20009 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:17:16.833029 20009 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:16.837100 20009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec: Generated new TSK 0
I20260620 07:17:16.837739 20009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:17:16.864039 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:37605
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:16.974680 20014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:16.974913 20014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:16.974961 20014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:16.978902 20014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:16.978953 20014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:16.979008 20014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:17:16.984001 20014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:37605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:16.984886 20014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:16.985885 20014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:16.995244 20022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:16.995253 20020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:16.995293 20019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:16.995569 20014 server_base.cc:1034] running on GCE node
I20260620 07:17:16.996031 20014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:16.996690 20014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:16.997879 20014 hybrid_clock.cc:648] HybridClock initialized: now 1781939836997850 us; error 38 us; skew 500 ppm
Jun 20 07:17:16 dist-test-slave-j663 krb5kdc[19907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939836, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:17.000685 20014 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:17:17.001611 20014 webserver.cc:460] Webserver started at http://127.11.123.129:44779/ using document root <none> and password file <none>
I20260620 07:17:17.002198 20014 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:17.002292 20014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:17.002519 20014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:17.004491 20014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "7702251725a54d629d4696f1ee9612a4"
format_stamp: "Formatted at 2026-06-20 07:17:16 on dist-test-slave-j663"
server_key: "a4525ae6b8f7482fb50cbf7ac31e8eb7"
server_key_iv: "710329fa35339e13bc64d651748ce875"
server_key_version: "encryptionkey@0"
I20260620 07:17:17.005048 20014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "7702251725a54d629d4696f1ee9612a4"
format_stamp: "Formatted at 2026-06-20 07:17:16 on dist-test-slave-j663"
server_key: "a4525ae6b8f7482fb50cbf7ac31e8eb7"
server_key_iv: "710329fa35339e13bc64d651748ce875"
server_key_version: "encryptionkey@0"
I20260620 07:17:17.008580 20014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:17:17.010906 20029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.012089 20014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:17.012218 20014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "7702251725a54d629d4696f1ee9612a4"
format_stamp: "Formatted at 2026-06-20 07:17:16 on dist-test-slave-j663"
server_key: "a4525ae6b8f7482fb50cbf7ac31e8eb7"
server_key_iv: "710329fa35339e13bc64d651748ce875"
server_key_version: "encryptionkey@0"
I20260620 07:17:17.012332 20014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:17.035295 20014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:17.038594 20014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:17.038796 20014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:17.039386 20014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:17.040205 20014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:17.040277 20014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.040343 20014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:17.040391 20014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.049340 20014 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:40245
I20260620 07:17:17.049386 20142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:40245 every 8 connection(s)
I20260620 07:17:17.050228 20014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939836, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:17.059790 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 20014
I20260620 07:17:17.059928 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:17:17.060209 11758 external_mini_cluster.cc:1428] Setting key 8e7870cc92dd62059f269550e934a49d
I20260620 07:17:17.063669 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:17.052000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:54629 (local address 127.11.123.190:37605)
0620 07:17:17.052135 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:17.052139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:17.053139 (+ 1000us) server_negotiation.cc:407] Connection header received
0620 07:17:17.054044 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:17.054047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:17.054106 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:17.054212 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:17.055345 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.055873 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.056990 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.057273 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.057385 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:17.059676 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:17.059679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:17.059681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:17.059705 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:17.061532 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:17.062152 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:17.062157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:17.062158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:17.062252 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:17.062571 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:17.062577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:17.062582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:17.062829 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:17.062934 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:17.063367 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 07:17:17.063522 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:17:17.064769 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:17.052350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:54629)
0620 07:17:17.052972 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:17.053010 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:17.053842 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:17.054377 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:17.054384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:17.054728 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:17.055179 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:17.055191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.056004 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.056008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:17.056856 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:17.056865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.057428 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.057432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:17.057595 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:17.058102 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:17.058117 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:17.059479 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:17.061689 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:17.061695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:17.061709 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:17.062048 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:17.062348 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:17.062351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:17.062352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:17.062478 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:17.062930 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:17.062934 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:17.063147 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 07:17:17.063295 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 07:17:17.063677 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":112,"threads_started":1}
I20260620 07:17:17.065974 20143 heartbeater.cc:347] Connected to a master server at 127.11.123.190:37605
I20260620 07:17:17.066249 20143 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:17.066819 20143 heartbeater.cc:511] Master 127.11.123.190:37605 requested a full tablet report, sending...
I20260620 07:17:17.068389 19946 ts_manager.cc:194] Registered new tserver with Master: 7702251725a54d629d4696f1ee9612a4 (127.11.123.129:40245)
I20260620 07:17:17.069571 19946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:54629
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:17:17.085968 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:37605
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:17.192009 20148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:17.192292 20148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:17.192348 20148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:17.196205 20148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:17.196251 20148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:17.196306 20148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:17:17.201258 20148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:37605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:17.202231 20148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:17.203159 20148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:17.212339 20154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:17.212505 20148 server_base.cc:1034] running on GCE node
W20260620 07:17:17.212447 20153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:17.212369 20156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:17.213035 20148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:17.213681 20148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:17.214910 20148 hybrid_clock.cc:648] HybridClock initialized: now 1781939837214881 us; error 38 us; skew 500 ppm
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939837, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:17.217752 20148 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:17:17.218598 20148 webserver.cc:460] Webserver started at http://127.11.123.130:42847/ using document root <none> and password file <none>
I20260620 07:17:17.219141 20148 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:17.219199 20148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:17.219417 20148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:17.221457 20148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "fca2fa7e53d94503a654fdb7b4a4058c"
format_stamp: "Formatted at 2026-06-20 07:17:17 on dist-test-slave-j663"
server_key: "e0c92e6ec69a366413bbbbb0bfcdb436"
server_key_iv: "2552ca61540c6ec58a45351a430f1528"
server_key_version: "encryptionkey@0"
I20260620 07:17:17.222030 20148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "fca2fa7e53d94503a654fdb7b4a4058c"
format_stamp: "Formatted at 2026-06-20 07:17:17 on dist-test-slave-j663"
server_key: "e0c92e6ec69a366413bbbbb0bfcdb436"
server_key_iv: "2552ca61540c6ec58a45351a430f1528"
server_key_version: "encryptionkey@0"
I20260620 07:17:17.225566 20148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:17:17.227846 20163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.228971 20148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:17.229130 20148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "fca2fa7e53d94503a654fdb7b4a4058c"
format_stamp: "Formatted at 2026-06-20 07:17:17 on dist-test-slave-j663"
server_key: "e0c92e6ec69a366413bbbbb0bfcdb436"
server_key_iv: "2552ca61540c6ec58a45351a430f1528"
server_key_version: "encryptionkey@0"
I20260620 07:17:17.229238 20148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:17.257141 20148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:17.260035 20148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:17.260231 20148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:17.260802 20148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:17.261642 20148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:17.261711 20148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.261776 20148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:17.261827 20148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.270386 20148 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:36991
I20260620 07:17:17.270421 20276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:36991 every 8 connection(s)
I20260620 07:17:17.271221 20148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:17:17.271643 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 20148
I20260620 07:17:17.271767 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:17:17.272094 11758 external_mini_cluster.cc:1428] Setting key cae30444ecb01c4e3991919a95e79e1c
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939837, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
I20260620 07:17:17.286347 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:17.273040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:43701 (local address 127.11.123.190:37605)
0620 07:17:17.273188 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:17.273194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:17.274380 (+ 1186us) server_negotiation.cc:407] Connection header received
0620 07:17:17.275695 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:17.275698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:17.275753 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:17.275828 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:17.277246 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.277855 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.279671 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.279917 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.280002 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:17.282715 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:17.282718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:17.282720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:17.282741 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:17.283974 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:17.284497 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:17.284500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:17.284501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:17.284637 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:17.285030 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:17.285034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:17.285036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:17.285335 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:17.285459 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:17.285952 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 07:17:17.286111 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:17:17.287101 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:17.273499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:43701)
0620 07:17:17.274217 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:17.274270 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:17.275468 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:17.276002 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:17.276009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:17.276399 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:17.277068 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:17.277118 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.277996 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.278002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:17.279540 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:17.279554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.280070 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.280076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:17.280283 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:17.280874 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:17.280898 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:17.282566 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:17.284072 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:17.284076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:17.284087 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:17.284373 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:17.284741 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:17.284748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:17.284753 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:17.284906 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:17.285446 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:17.285450 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:17.285652 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 07:17:17.285792 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 07:17:17.286039 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":107,"threads_started":1}
I20260620 07:17:17.288369 20277 heartbeater.cc:347] Connected to a master server at 127.11.123.190:37605
I20260620 07:17:17.288625 20277 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:17.289243 20277 heartbeater.cc:511] Master 127.11.123.190:37605 requested a full tablet report, sending...
I20260620 07:17:17.290406 19946 ts_manager.cc:194] Registered new tserver with Master: fca2fa7e53d94503a654fdb7b4a4058c (127.11.123.130:36991)
I20260620 07:17:17.291255 19946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:43701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:17:17.299817 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:37605
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:17.410972 20282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:17.411204 20282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:17.411278 20282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:17.415298 20282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:17.415349 20282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:17.415452 20282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:17:17.420753 20282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:37605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:17.421751 20282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:17.422703 20282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:17.431944 20290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:17.431990 20288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:17.431968 20287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:17.432541 20282 server_base.cc:1034] running on GCE node
I20260620 07:17:17.432991 20282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:17.433650 20282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:17.434938 20282 hybrid_clock.cc:648] HybridClock initialized: now 1781939837434917 us; error 29 us; skew 500 ppm
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939837, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:17.437546 20282 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:17:17.438383 20282 webserver.cc:460] Webserver started at http://127.11.123.131:33173/ using document root <none> and password file <none>
I20260620 07:17:17.438932 20282 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:17.438997 20282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:17.439203 20282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:17.441260 20282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "e610b3112ef5434c83045cf96cb5bc17"
format_stamp: "Formatted at 2026-06-20 07:17:17 on dist-test-slave-j663"
server_key: "c13425b21d45f88f787ac7520d3cba4e"
server_key_iv: "78c8ea52ac881c812b217ecd07685027"
server_key_version: "encryptionkey@0"
I20260620 07:17:17.441821 20282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "e610b3112ef5434c83045cf96cb5bc17"
format_stamp: "Formatted at 2026-06-20 07:17:17 on dist-test-slave-j663"
server_key: "c13425b21d45f88f787ac7520d3cba4e"
server_key_iv: "78c8ea52ac881c812b217ecd07685027"
server_key_version: "encryptionkey@0"
I20260620 07:17:17.445391 20282 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:17:17.447698 20297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.448911 20282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:17.449043 20282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "e610b3112ef5434c83045cf96cb5bc17"
format_stamp: "Formatted at 2026-06-20 07:17:17 on dist-test-slave-j663"
server_key: "c13425b21d45f88f787ac7520d3cba4e"
server_key_iv: "78c8ea52ac881c812b217ecd07685027"
server_key_version: "encryptionkey@0"
I20260620 07:17:17.449200 20282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:17.464776 20282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:17.467948 20282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:17.468154 20282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:17.468739 20282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:17.469630 20282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:17.469707 20282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.469777 20282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:17.469821 20282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.478693 20282 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:34813
I20260620 07:17:17.478713 20410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:34813 every 8 connection(s)
I20260620 07:17:17.479527 20282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:17:17.485895 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 20282
I20260620 07:17:17.486040 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:17:17.486359 11758 external_mini_cluster.cc:1428] Setting key eb1e0f98376fd2a55250ed7827169064
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939837, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:17.493516 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:17.481380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:44289 (local address 127.11.123.190:37605)
0620 07:17:17.481542 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:17.481547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:17.482713 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 07:17:17.483489 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:17.483492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:17.483540 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:17.483626 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:17.484736 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.485444 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.487020 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.487342 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.487456 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:17.490053 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:17.490057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:17.490060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:17.490087 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:17.491487 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:17.492003 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:17.492007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:17.492009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:17.492094 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:17.492439 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:17.492443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:17.492445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:17.492622 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:17.492731 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:17.493221 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 07:17:17.493351 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:17:17.494261 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:17.481868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:44289)
0620 07:17:17.482553 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:17.482592 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:17.483297 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:17.483778 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:17.483784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:17.484126 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:17.484589 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:17.484603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.485684 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.485690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:17.486911 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:17.486920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.487489 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.487492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:17.487712 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:17.488380 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:17.488395 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:17.489880 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:17.491609 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:17.491615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:17.491627 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:17.491895 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:17.492195 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:17.492198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:17.492200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:17.492343 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:17.492728 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:17.492734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:17.492962 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 07:17:17.493181 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 07:17:17.493488 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":127,"threads_started":1}
I20260620 07:17:17.495333 20411 heartbeater.cc:347] Connected to a master server at 127.11.123.190:37605
I20260620 07:17:17.495594 20411 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:17.496055 20411 heartbeater.cc:511] Master 127.11.123.190:37605 requested a full tablet report, sending...
I20260620 07:17:17.497028 19946 ts_manager.cc:194] Registered new tserver with Master: e610b3112ef5434c83045cf96cb5bc17 (127.11.123.131:34813)
I20260620 07:17:17.497622 19946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:44289
I20260620 07:17:17.500133 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab.
Entry for principal oryx/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab.
WARNING: no policy specified for oryx/127.11.123.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab.
Entry for principal oryx/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab.
WARNING: no policy specified for oryx/127.11.123.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab.
Entry for principal oryx/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab.
I20260620 07:17:17.569665 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 20014
I20260620 07:17:17.575205 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 20148
I20260620 07:17:17.580564 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 20282
I20260620 07:17:17.585670 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 19920
I20260620 07:17:17.591554 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:37605
--webserver_interface=127.11.123.190
--webserver_port=44737
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:37605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:17.696848 20420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:17.697162 20420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:17.697269 20420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:17.701215 20420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:17.701298 20420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:17.701333 20420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:17.701370 20420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:17.701402 20420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:17:17.706372 20420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:37605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:37605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=44737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:17.707360 20420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:17.708323 20420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:17.716467 20426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:17.716434 20428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:17.716526 20420 server_base.cc:1034] running on GCE node
W20260620 07:17:17.716434 20425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:17.717216 20420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:17.718230 20420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:17.719393 20420 hybrid_clock.cc:648] HybridClock initialized: now 1781939837719380 us; error 24 us; skew 500 ppm
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939837, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:17.722157 20420 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:17:17.723035 20420 webserver.cc:460] Webserver started at http://127.11.123.190:44737/ using document root <none> and password file <none>
I20260620 07:17:17.723567 20420 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:17.723667 20420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:17.727423 20420 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:17:17.729476 20435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.730728 20420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:17.730949 20420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "b2f6f2b23e1e45a287268deb3b3c6fec"
format_stamp: "Formatted at 2026-06-20 07:17:16 on dist-test-slave-j663"
server_key: "8d619d75ab72ffd39cb31e3eda3bc2c2"
server_key_iv: "1ee4241876eea3cc62da408542c79879"
server_key_version: "encryptionkey@0"
I20260620 07:17:17.731477 20420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:17.752959 20420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:17.755628 20420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:17.755827 20420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:17.762892 20420 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:37605
I20260620 07:17:17.762915 20487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:37605 every 8 connection(s)
I20260620 07:17:17.763877 20420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:17.767069 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 20420
I20260620 07:17:17.767509 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:40245
--local_ip_for_outbound_sockets=127.11.123.129
--tserver_master_addrs=127.11.123.190:37605
--webserver_port=44779
--webserver_interface=127.11.123.129
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.123.129
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:17:17.767983 20488 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:17:17.770053 20488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec: Bootstrap starting.
I20260620 07:17:17.792814 20488 log.cc:824] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:17.801110 20488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:17:17.801506 20488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec: Bootstrap complete.
I20260620 07:17:17.807055 20488 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37605 } }
I20260620 07:17:17.807551 20488 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f6f2b23e1e45a287268deb3b3c6fec, State: Initialized, Role: FOLLOWER
I20260620 07:17:17.808091 20488 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37605 } }
I20260620 07:17:17.808233 20488 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:17.808295 20488 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:17.808387 20488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:17:17.810915 20488 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37605 } }
I20260620 07:17:17.811297 20488 leader_election.cc:302] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2f6f2b23e1e45a287268deb3b3c6fec; no voters:
I20260620 07:17:17.811646 20488 leader_election.cc:288] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:17:17.812038 20493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:17:17.812338 20493 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [term 2 LEADER]: Becoming Leader. State: Replica: b2f6f2b23e1e45a287268deb3b3c6fec, State: Running, Role: LEADER
I20260620 07:17:17.812690 20493 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37605 } }
I20260620 07:17:17.812861 20488 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:17.814033 20494 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f6f2b23e1e45a287268deb3b3c6fec. Latest consensus state: current_term: 2 leader_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37605 } } }
I20260620 07:17:17.814146 20494 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:17.814817 20492 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f6f2b23e1e45a287268deb3b3c6fec" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37605 } } }
I20260620 07:17:17.814909 20492 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:17.815163 20501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:17.818338 20501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:17.818979 20501 catalog_manager.cc:1259] Loaded cluster ID: 1113fa9c90a64e71b6f0c9b2c96d6a33
I20260620 07:17:17.819039 20501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:17.821383 20501 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:17.822011 20501 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b2f6f2b23e1e45a287268deb3b3c6fec: Loaded TSK: 0
I20260620 07:17:17.823221 20501 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:17:17.893821 20490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:17.894094 20490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:17.894167 20490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:17.898800 20490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:17.898888 20490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:17.898976 20490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:17:17.904218 20490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.129.keytab
--principal=oryx/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:40245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=44779
--tserver_master_addrs=127.11.123.190:37605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:17.905248 20490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:17.906216 20490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:17.916589 20516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:17.916586 20518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:17.916694 20515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:17.917115 20490 server_base.cc:1034] running on GCE node
I20260620 07:17:17.917520 20490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:17.918207 20490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:17.919376 20490 hybrid_clock.cc:648] HybridClock initialized: now 1781939837919362 us; error 37 us; skew 500 ppm
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939837, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:17.922199 20490 init.cc:377] Logged in from keytab as oryx/127.11.123.129@KRBTEST.COM (short username oryx)
I20260620 07:17:17.923182 20490 webserver.cc:460] Webserver started at http://127.11.123.129:44779/ using document root <none> and password file <none>
I20260620 07:17:17.923813 20490 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:17.923918 20490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:17.928047 20490 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:17:17.930405 20525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.931763 20490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:17:17.931893 20490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "7702251725a54d629d4696f1ee9612a4"
format_stamp: "Formatted at 2026-06-20 07:17:16 on dist-test-slave-j663"
server_key: "a4525ae6b8f7482fb50cbf7ac31e8eb7"
server_key_iv: "710329fa35339e13bc64d651748ce875"
server_key_version: "encryptionkey@0"
I20260620 07:17:17.932451 20490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:17.943056 20490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:17.946094 20490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:17.946291 20490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:17.946906 20490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:17.947767 20490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:17.947840 20490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.947908 20490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:17.947960 20490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:17.957921 20490 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:40245
I20260620 07:17:17.957983 20638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:40245 every 8 connection(s)
I20260620 07:17:17.958922 20490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:17:17.964305 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 20490
I20260620 07:17:17.964790 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:36991
--local_ip_for_outbound_sockets=127.11.123.130
--tserver_master_addrs=127.11.123.190:37605
--webserver_port=42847
--webserver_interface=127.11.123.130
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.123.130
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:17.972791 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:17.961189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:59013)
0620 07:17:17.961848 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:17.961886 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:17.962624 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:17.963614 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:17.963620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:17.963988 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:17.964511 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:17.964528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.966439 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.966446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:17.967897 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:17.967910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.968623 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.968629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:17.968850 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:17.969486 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:17.969515 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:17.972095 (+ 2580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":128,"threads_started":1}
W20260620 07:17:17.973364 20641 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:17.961145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:59013 (local address 127.11.123.190:37605)
0620 07:17:17.961597 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:17.961616 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:17.962015 (+ 399us) server_negotiation.cc:407] Connection header received
0620 07:17:17.962952 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:17.962963 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:17.963243 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:17.963502 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:17.964680 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.966293 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.968017 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.968461 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.968588 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:17.972397 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:59013: BlockingRecv error: recv got EOF from 127.11.123.129:59013 (error 108)
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":99,"threads_started":1}
W20260620 07:17:17.973481 20639 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:37605 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.123.190:37605: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:17.979327 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:17.974103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:49931)
0620 07:17:17.974295 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:17.974315 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:17.974375 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:17.974619 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:17.974621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:17.974743 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:17.974829 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:17.974833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.975898 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.975900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:17.976424 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:17.976430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.976981 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.976983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:17.977059 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:17.977570 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:17.977581 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:17.979245 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129}
W20260620 07:17:17.979491 20641 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:17.974233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:49931 (local address 127.11.123.190:37605)
0620 07:17:17.974380 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:17.974387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:17.974403 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:17:17.974449 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:17.974451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:17.974505 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:17.974603 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:17.974938 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.975799 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.976531 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.976875 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.976969 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:17.979392 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:49931: BlockingRecv error: recv got EOF from 127.11.123.129:49931 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:17 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:17.984488 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:17.979904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:37685)
0620 07:17:17.980028 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:17.980043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:17.980123 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:17.980373 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:17.980376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:17.980520 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:17.980581 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:17.980586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.981331 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.981337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:17.981830 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:17.981837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.982273 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.982276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:17.982351 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:17.982810 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:17.982821 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:17.984363 (+ 1542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:17:17.984642 20639 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:17:17.984730 20641 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:17.979978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:37685 (local address 127.11.123.190:37605)
0620 07:17:17.980086 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:17.980091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:17.980104 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:17:17.980232 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:17.980235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:17.980273 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:17.980361 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:17.980690 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.981233 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.981944 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:17.982168 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:17.982274 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:17.984467 (+ 2193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:37685: BlockingRecv error: recv got EOF from 127.11.123.129:37685 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 07:17:18.076809 20643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:18.077080 20643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:18.077190 20643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:18.082086 20643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:18.082166 20643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:18.082265 20643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:17:18.087424 20643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.130.keytab
--principal=oryx/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:36991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=42847
--tserver_master_addrs=127.11.123.190:37605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:18.088399 20643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:18.089425 20643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:18.099051 20648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:18.099071 20649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:18.099071 20651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:18.099213 20643 server_base.cc:1034] running on GCE node
I20260620 07:17:18.099647 20643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:18.100303 20643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:18.101493 20643 hybrid_clock.cc:648] HybridClock initialized: now 1781939838101475 us; error 32 us; skew 500 ppm
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939838, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:18.104247 20643 init.cc:377] Logged in from keytab as oryx/127.11.123.130@KRBTEST.COM (short username oryx)
I20260620 07:17:18.105216 20643 webserver.cc:460] Webserver started at http://127.11.123.130:42847/ using document root <none> and password file <none>
I20260620 07:17:18.105767 20643 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:18.105875 20643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:18.109650 20643 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 07:17:18.111774 20658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:18.112988 20643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:18.113153 20643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "fca2fa7e53d94503a654fdb7b4a4058c"
format_stamp: "Formatted at 2026-06-20 07:17:17 on dist-test-slave-j663"
server_key: "e0c92e6ec69a366413bbbbb0bfcdb436"
server_key_iv: "2552ca61540c6ec58a45351a430f1528"
server_key_version: "encryptionkey@0"
I20260620 07:17:18.113687 20643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:18.140602 20643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:18.143697 20643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:18.143901 20643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:18.144484 20643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:18.145328 20643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:18.145395 20643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:18.145491 20643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:18.145541 20643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:18.154506 20643 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:36991
I20260620 07:17:18.154536 20771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:36991 every 8 connection(s)
I20260620 07:17:18.155434 20643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:17:18.161599 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 20643
I20260620 07:17:18.162019 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:34813
--local_ip_for_outbound_sockets=127.11.123.131
--tserver_master_addrs=127.11.123.190:37605
--webserver_port=33173
--webserver_interface=127.11.123.131
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.123.131
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:17:18.169701 20641 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:18.157178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:36225 (local address 127.11.123.190:37605)
0620 07:17:18.157313 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:18.157317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:18.158296 (+ 979us) server_negotiation.cc:407] Connection header received
0620 07:17:18.159170 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:18.159174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:18.159237 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:18.159347 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:18.160482 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.161011 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.162242 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.162466 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.162595 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.169559 (+ 6964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:36225: BlockingRecv error: recv got EOF from 127.11.123.130:36225 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:17:18.170576 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:18.157530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:36225)
0620 07:17:18.158147 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:18.158186 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:18.158957 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:18.159519 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:18.159525 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:18.159878 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:18.160333 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.160346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.161154 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.161158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.162104 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.162135 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.162640 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.162643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.162810 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.166159 (+ 3349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:18.166189 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:18.169340 (+ 3151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":101,"threads_started":1}
W20260620 07:17:18.171166 20772 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:37605 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.123.190:37605: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:18.177371 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:18.171655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:51525)
0620 07:17:18.171829 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:18.171846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:18.171941 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:18.172205 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:18.172211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:18.172416 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:18.172573 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.172582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.173388 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.173392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.174291 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.174301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.174715 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.174718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.174814 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.175258 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:18.175272 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:18.177255 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 07:17:18.177520 20641 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:18.171787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:51525 (local address 127.11.123.190:37605)
0620 07:17:18.171930 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:18.171933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:18.171948 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:17:18.172047 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:18.172050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:18.172088 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:18.172155 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:18.172694 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.173290 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.174411 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.174621 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.174695 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.177378 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:51525: BlockingRecv error: recv got EOF from 127.11.123.130:51525 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:18.182894 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:18.178032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:43191)
0620 07:17:18.178125 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:18.178138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:18.178205 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:18.178421 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:18.178424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:18.178574 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:18.178652 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.178657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.179299 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.179305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.180229 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.180239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.180643 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.180647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.180725 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.181131 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:18.181145 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:18.182787 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 07:17:18.182993 20641 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:18.178097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:43191 (local address 127.11.123.190:37605)
0620 07:17:18.178205 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:18.178208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:18.178220 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:17:18.178290 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:18.178292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:18.178323 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:18.178385 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:18.178749 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.179201 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.180341 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.180543 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.180618 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.182888 (+ 2270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:43191: BlockingRecv error: recv got EOF from 127.11.123.130:43191 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 07:17:18.183282 20772 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:17:18.273092 20775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:18.273335 20775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:18.273375 20775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:18.277782 20775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:18.277834 20775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:18.277889 20775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:17:18.282872 20775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/krb5kdc/oryx_127.11.123.131.keytab
--principal=oryx/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:34813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=33173
--tserver_master_addrs=127.11.123.190:37605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:18.283766 20775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:18.284678 20775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:18.293713 20781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:18.293807 20775 server_base.cc:1034] running on GCE node
W20260620 07:17:18.293720 20780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:18.293730 20783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:18.294384 20775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:18.295011 20775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:18.296190 20775 hybrid_clock.cc:648] HybridClock initialized: now 1781939838296162 us; error 39 us; skew 500 ppm
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939838, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:18.298849 20775 init.cc:377] Logged in from keytab as oryx/127.11.123.131@KRBTEST.COM (short username oryx)
I20260620 07:17:18.299600 20775 webserver.cc:460] Webserver started at http://127.11.123.131:33173/ using document root <none> and password file <none>
I20260620 07:17:18.300094 20775 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:18.300163 20775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:18.303763 20775 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:17:18.305703 20790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:18.306883 20775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:18.307001 20775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "e610b3112ef5434c83045cf96cb5bc17"
format_stamp: "Formatted at 2026-06-20 07:17:17 on dist-test-slave-j663"
server_key: "c13425b21d45f88f787ac7520d3cba4e"
server_key_iv: "78c8ea52ac881c812b217ecd07685027"
server_key_version: "encryptionkey@0"
I20260620 07:17:18.307525 20775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:18.323463 20775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:18.326346 20775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:18.326493 20775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:18.327013 20775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:18.327872 20775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:18.327917 20775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:18.327956 20775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:18.327971 20775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:18.336651 20775 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:34813
I20260620 07:17:18.336674 20903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:34813 every 8 connection(s)
I20260620 07:17:18.337700 20775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:17:18.338033 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 20775
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:18.347760 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:18.338747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.11.123.190:37605)
0620 07:17:18.338889 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:18.338893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:18.339101 (+ 208us) server_negotiation.cc:407] Connection header received
0620 07:17:18.339199 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:18.339202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:18.339256 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:18.339345 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:18.340281 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.340796 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.341798 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.342140 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.342256 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.343771 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:18.343797 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:18.343815 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:18.343849 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:18.345630 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:18.346158 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:18.346169 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:18.346173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:18.346286 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:18.346573 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:18.346577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:18.346579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:18.346890 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:18.347083 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:18.347362 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 07:17:18.347595 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:17:18.349411 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:18.340243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:50599)
0620 07:17:18.340944 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:18.340989 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:18.341745 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:18.342186 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:18.342193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:18.342506 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:18.343090 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.343104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.344080 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.344084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.344930 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.344939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.345476 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.345480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.345654 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.346130 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:18.346145 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:18.348471 (+ 2326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":131,"threads_started":1}
W20260620 07:17:18.349678 20907 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:18.339971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:50599 (local address 127.11.123.190:37605)
0620 07:17:18.340200 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:18.340203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:18.341102 (+ 899us) server_negotiation.cc:407] Connection header received
0620 07:17:18.341913 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:18.341916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:18.341960 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:18.342045 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:18.343242 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.343949 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.345064 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.345328 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.345439 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.349409 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:50599: BlockingRecv error: recv got EOF from 127.11.123.131:50599 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
W20260620 07:17:18.349932 20904 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:37605 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.123.190:37605: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:18.355090 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:18.350331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:37531)
0620 07:17:18.350719 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:18.350731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:18.350834 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:18.351055 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:18.351058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:18.351213 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:18.351293 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.351298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.352005 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.352009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.352541 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.352549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.352993 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.352997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.353115 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.353474 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:18.353484 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:18.355015 (+ 1531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333}
W20260620 07:17:18.355261 20907 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:18.350382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:37531 (local address 127.11.123.190:37605)
0620 07:17:18.350474 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:18.350477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:18.350799 (+ 322us) server_negotiation.cc:407] Connection header received
0620 07:17:18.350914 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:18.350917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:18.350954 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:18.351041 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:18.351402 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.351890 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.352661 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.352886 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.352991 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.355151 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:37531: BlockingRecv error: recv got EOF from 127.11.123.131:37531 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:18.359635 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:18.355514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:33547)
0620 07:17:18.355611 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:18.355624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:18.355692 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:18.355896 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:18.355898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:18.356018 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:18.356080 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.356084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.356696 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.356698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.357266 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.357273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.357648 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.357651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.357705 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.358117 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:18.358127 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:18.359560 (+ 1433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 07:17:18.359771 20907 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:18.355604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:33547 (local address 127.11.123.190:37605)
0620 07:17:18.355722 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:18.355725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:18.355736 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:17:18.355762 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:18.355764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:18.355799 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:18.355862 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:18.356169 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.356616 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.357356 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.357553 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.357629 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.359665 (+ 2036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:33547: BlockingRecv error: recv got EOF from 127.11.123.131:33547 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 07:17:18.359956 20904 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:17:18.960076 20634 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.11.123.129:40245
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:18 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:18.990882 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:18.985255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:33157)
0620 07:17:18.985512 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:18.985526 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:18.985610 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:18.985877 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:18.985880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:18.986064 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:18.986179 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.986184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.986898 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.986901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.987513 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:18.987528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.988218 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.988224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:18.988334 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.988876 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:18.988895 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:18.990771 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
W20260620 07:17:18.991035 20910 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:18.985381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:33157 (local address 127.11.123.190:37605)
0620 07:17:18.985603 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:18.985607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:18.985620 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:17:18.985706 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:18.985708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:18.985756 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:18.985841 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:18.986290 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.986806 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.987655 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:18.988072 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:18.988205 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:18.990936 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:33157: BlockingRecv error: recv got EOF from 127.11.123.129:33157 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
Jun 20 07:17:19 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:19 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:19.190059 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:19.183970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:33057)
0620 07:17:19.184262 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:19.184277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:19.184438 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:19.184742 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:19.184746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:19.185018 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:19.185170 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:19.185177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:19.186011 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:19.186017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:19.186655 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:19.186664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:19.187176 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:19.187180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:19.187285 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:19.187905 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:19.187918 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:19.189878 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":77,"threads_started":1}
W20260620 07:17:19.190140 20910 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:19.184028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:33057 (local address 127.11.123.190:37605)
0620 07:17:19.184196 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:19.184201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:19.184367 (+ 166us) server_negotiation.cc:407] Connection header received
0620 07:17:19.184556 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:19.184561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:19.184618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:19.184716 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:19.185300 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:19.185877 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:19.186786 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:19.187043 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:19.187162 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:19.190018 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:33057: BlockingRecv error: recv got EOF from 127.11.123.130:33057 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 07:17:19 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:19 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:19.365804 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:19.360569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:53371)
0620 07:17:19.360840 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:19.360858 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:19.360960 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:19.361181 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:19.361183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:19.361366 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:19.361462 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:19.361468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:19.362193 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:19.362196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:19.362765 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:19.362773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:19.363191 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:19.363193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:19.363254 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:19.363814 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:19.363825 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:19.365642 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
W20260620 07:17:19.365936 20910 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:19.360620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:53371 (local address 127.11.123.190:37605)
0620 07:17:19.360726 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:19.360730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:19.360916 (+ 186us) server_negotiation.cc:407] Connection header received
0620 07:17:19.361027 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:19.361029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:19.361090 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:19.361153 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:19.361563 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:19.362072 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:19.362880 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:19.363095 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:19.363174 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:19.365804 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:53371: BlockingRecv error: recv got EOF from 127.11.123.131:53371 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 07:17:19 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:19 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:19.997258 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:19.991758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:35113)
0620 07:17:19.992084 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:19.992097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:19.992197 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:19.992438 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:19.992440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:19.992613 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:19.992710 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:19.992715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:19.993455 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:19.993457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:19.994015 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:19.994022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:19.994481 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:19.994484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:19.994545 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:19.995114 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:19.995126 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:19.997057 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":78,"threads_started":1}
W20260620 07:17:19.997484 20914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:19.991815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:35113 (local address 127.11.123.190:37605)
0620 07:17:19.992109 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:19.992112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:19.992169 (+ 57us) server_negotiation.cc:407] Connection header received
0620 07:17:19.992285 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:19.992287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:19.992329 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:19.992404 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:19.992807 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:19.993355 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:19.994138 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:19.994379 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:19.994466 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:19.997328 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:35113: BlockingRecv error: recv got EOF from 127.11.123.129:35113 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
Jun 20 07:17:20 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:20 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:20.196849 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:20.190894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:46889)
0620 07:17:20.191214 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:20.191233 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:20.191371 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:20.191672 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:20.191676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:20.191883 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:20.191997 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:20.192003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:20.192776 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:20.192782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:20.193440 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:20.193450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:20.193900 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:20.193904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:20.193999 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:20.194488 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:20.194499 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:20.196683 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
W20260620 07:17:20.196947 20914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:20.191003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:46889 (local address 127.11.123.190:37605)
0620 07:17:20.191177 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:20.191181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:20.191310 (+ 129us) server_negotiation.cc:407] Connection header received
0620 07:17:20.191493 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:20.191498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:20.191555 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:20.191644 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:20.192125 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:20.192657 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:20.193563 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:20.193788 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:20.193899 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:20.196820 (+ 2921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:46889: BlockingRecv error: recv got EOF from 127.11.123.130:46889 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 07:17:20 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:20 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:20.372078 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:20.366607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:47209)
0620 07:17:20.366887 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:20.366900 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:20.367005 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:20.367240 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:20.367242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:20.367413 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:20.367506 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:20.367511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:20.368300 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:20.368303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:20.369174 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:20.369185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:20.369653 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:20.369656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:20.369717 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:20.370188 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:20.370199 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:20.371938 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
W20260620 07:17:20.372193 20914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:20.366779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:47209 (local address 127.11.123.190:37605)
0620 07:17:20.366940 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:20.366944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:20.366957 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:17:20.367088 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:20.367091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:20.367133 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:20.367200 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:20.367598 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:20.368177 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:20.369295 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:20.369550 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:20.369632 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:20.372063 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:47209: BlockingRecv error: recv got EOF from 127.11.123.131:47209 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 07:17:20 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:20 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:21.003608 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:20.998161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:58437)
0620 07:17:20.998440 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:20.998454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:20.998541 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:20.998790 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:20.998792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:20.998999 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:20.999098 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:20.999103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:20.999884 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:20.999887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:21.000433 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:21.000440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:21.000872 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:21.000875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:21.000937 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:21.001520 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:21.001532 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:21.003423 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
W20260620 07:17:21.003664 20918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:20.998231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:58437 (local address 127.11.123.190:37605)
0620 07:17:20.998497 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:20.998501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:20.998515 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:17:20.998633 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:20.998635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:20.998679 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:20.998761 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:20.999204 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:20.999751 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:21.000532 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:21.000748 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:21.000823 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:21.003539 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:58437: BlockingRecv error: recv got EOF from 127.11.123.129:58437 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
Jun 20 07:17:21 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:21 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:21.202930 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:21.197652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:33461)
0620 07:17:21.198020 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:21.198034 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:21.198143 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:21.198335 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:21.198337 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:21.198509 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:21.198601 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:21.198606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:21.199337 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:21.199339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:21.199931 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:21.199939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:21.200324 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:21.200326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:21.200385 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:21.200912 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:21.200933 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:21.202748 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":73,"threads_started":1}
W20260620 07:17:21.202967 20918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:21.197710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:33461 (local address 127.11.123.190:37605)
0620 07:17:21.197843 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:21.197850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:21.198117 (+ 267us) server_negotiation.cc:407] Connection header received
0620 07:17:21.198208 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:21.198210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:21.198258 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:21.198323 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:21.198690 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:21.199243 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:21.200028 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:21.200236 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:21.200308 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:21.202864 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:33461: BlockingRecv error: recv got EOF from 127.11.123.130:33461 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 07:17:21 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:21 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:21.378332 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:21.372769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:58703)
0620 07:17:21.373022 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:21.373034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:21.373151 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:21.373382 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:21.373384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:21.373556 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:21.373650 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:21.373656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:21.374406 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:21.374409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:21.375122 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:21.375144 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:21.375638 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:21.375640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:21.375702 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:21.376230 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:21.376256 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:21.378219 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
W20260620 07:17:21.378499 20918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:21.372842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:58703 (local address 127.11.123.190:37605)
0620 07:17:21.372980 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:21.372984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:21.373101 (+ 117us) server_negotiation.cc:407] Connection header received
0620 07:17:21.373228 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:21.373230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:21.373275 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:21.373349 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:21.373744 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:21.374315 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:21.375272 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:21.375543 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:21.375622 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:21.378340 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:58703: BlockingRecv error: recv got EOF from 127.11.123.131:58703 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 07:17:22 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:22 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:22.009717 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:22.004352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:56793)
0620 07:17:22.004625 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:22.004642 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:22.004744 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:22.004973 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:22.004975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:22.005174 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:22.005273 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:22.005278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:22.006036 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:22.006038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:22.006587 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:22.006594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:22.007011 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:22.007013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:22.007074 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:22.007617 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:22.007629 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:22.009543 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
W20260620 07:17:22.009784 20922 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:22.004472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:56793 (local address 127.11.123.190:37605)
0620 07:17:22.004716 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:22.004719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:22.004734 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:17:22.004822 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:22.004824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:22.004865 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:22.004934 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:22.005406 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:22.005942 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:22.006682 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:22.006917 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:22.006994 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:22.009664 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:56793: BlockingRecv error: recv got EOF from 127.11.123.129:56793 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
Jun 20 07:17:22 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:22 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:22.209142 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:22.203693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:47985)
0620 07:17:22.203962 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:22.203976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:22.204063 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:22.204307 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:22.204310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:22.204493 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:22.204591 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:22.204596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:22.205408 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:22.205411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:22.205949 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:22.205957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:22.206389 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:22.206392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:22.206459 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:22.206956 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:22.206967 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:22.208972 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
W20260620 07:17:22.209303 20922 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:22.203926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:47985 (local address 127.11.123.190:37605)
0620 07:17:22.204068 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:22.204071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:22.204084 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:17:22.204136 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:22.204138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:22.204180 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:22.204255 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:22.204700 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:22.205314 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:22.206066 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:22.206290 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:22.206372 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:22.209151 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:47985: BlockingRecv error: recv got EOF from 127.11.123.130:47985 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 07:17:22 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:22 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:22.384938 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:22.379192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:52715)
0620 07:17:22.379519 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:22.379532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:22.379668 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:22.379966 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:22.379969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:22.380205 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:22.380334 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:22.380340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:22.381161 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:22.381165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:22.381717 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:22.381724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:22.382198 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:22.382202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:22.382288 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:22.382975 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:22.382986 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:22.384769 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
W20260620 07:17:22.385028 20922 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:22.379308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:52715 (local address 127.11.123.190:37605)
0620 07:17:22.379466 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:22.379471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:22.379605 (+ 134us) server_negotiation.cc:407] Connection header received
0620 07:17:22.379795 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:22.379798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:22.379851 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:22.379948 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:22.380459 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:22.381009 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:22.381840 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:22.382077 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:22.382185 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:22.384896 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:52715: BlockingRecv error: recv got EOF from 127.11.123.131:52715 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 07:17:23 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:23 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:23.016090 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:23.010542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:37401)
0620 07:17:23.010793 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:23.010810 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:23.010939 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:23.011200 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:23.011202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:23.011381 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:23.011492 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:23.011497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:23.012339 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:23.012341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:23.012947 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:23.012956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:23.013467 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:23.013473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:23.013571 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:23.014168 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:23.014181 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:23.015975 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
W20260620 07:17:23.016235 20926 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:23.010582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:37401 (local address 127.11.123.190:37605)
0620 07:17:23.010832 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:23.010837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:23.010862 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:17:23.011036 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:23.011039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:23.011083 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:23.011167 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:23.011597 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:23.012240 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:23.013054 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:23.013341 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:23.013423 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:23.016106 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:37401: BlockingRecv error: recv got EOF from 127.11.123.129:37401 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":104,"threads_started":1}
Jun 20 07:17:23 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:23 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:23.221298 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:23.210239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:55167)
0620 07:17:23.210563 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:23.210582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:23.210675 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:23.210905 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:23.210907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:23.211084 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:23.211176 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:23.211181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:23.212038 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:23.212043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:23.212959 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:23.212969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:23.213402 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:23.213408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:23.213487 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:23.219319 (+ 5832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:23.219332 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:23.221177 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":126,"threads_started":1}
W20260620 07:17:23.221410 20926 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:23.210440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:55167 (local address 127.11.123.190:37605)
0620 07:17:23.210575 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:23.210579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:23.210653 (+ 74us) server_negotiation.cc:407] Connection header received
0620 07:17:23.210753 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:23.210755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:23.210795 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:23.210866 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:23.211268 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:23.211904 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:23.213063 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:23.213289 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:23.213369 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:23.221293 (+ 7924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:55167: BlockingRecv error: recv got EOF from 127.11.123.130:55167 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 07:17:23 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:23 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:23.391181 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:23.385889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:33755)
0620 07:17:23.386184 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:23.386197 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:23.386291 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:23.386497 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:23.386499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:23.386674 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:23.386763 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:23.386768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:23.387456 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:23.387458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:23.388003 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:23.388010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:23.388488 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:23.388492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:23.388558 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:23.389140 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:23.389152 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:23.391068 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":122,"threads_started":1}
W20260620 07:17:23.391351 20926 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:23.385967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:33755 (local address 127.11.123.190:37605)
0620 07:17:23.386108 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:23.386113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:23.386251 (+ 138us) server_negotiation.cc:407] Connection header received
0620 07:17:23.386356 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:23.386358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:23.386399 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:23.386464 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:23.386859 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:23.387371 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:23.388121 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:23.388361 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:23.388448 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:23.391217 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:33755: BlockingRecv error: recv got EOF from 127.11.123.131:33755 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 07:17:24 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:24 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:24.022743 20929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:24.016966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:48595)
0620 07:17:24.017222 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:24.017238 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:24.017421 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:24.017694 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:24.017697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:24.017927 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:24.018027 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:24.018033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:24.018799 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:24.018802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:24.019423 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:24.019433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:24.019869 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:24.019872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:24.019941 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:24.020581 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:24.020604 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:24.022632 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
W20260620 07:17:24.022948 20930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:24.017037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:48595 (local address 127.11.123.190:37605)
0620 07:17:24.017366 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:24.017378 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:24.017393 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:17:24.017519 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:24.017522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:24.017564 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:24.017639 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:24.018127 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:24.018646 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:24.019530 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:24.019762 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:24.019837 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:24.022791 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:48595: BlockingRecv error: recv got EOF from 127.11.123.129:48595 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":146,"threads_started":1}
Jun 20 07:17:24 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:24 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:24.227598 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:24.222252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:38683)
0620 07:17:24.222507 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:24.222526 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:24.222660 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:24.222929 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:24.222933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:24.223150 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:24.223244 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:24.223250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:24.224029 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:24.224032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:24.224615 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:24.224623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:24.225132 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:24.225135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:24.225222 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:24.225710 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:24.225721 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:24.227487 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
W20260620 07:17:24.227767 20930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:24.222271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:38683 (local address 127.11.123.190:37605)
0620 07:17:24.222420 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:24.222427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:24.222639 (+ 212us) server_negotiation.cc:407] Connection header received
0620 07:17:24.222753 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:24.222758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:24.222817 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:24.222898 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:24.223368 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:24.223898 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:24.224740 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:24.224991 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:24.225127 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:24.227614 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:38683: BlockingRecv error: recv got EOF from 127.11.123.130:38683 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 07:17:24 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:24 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:24.397536 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:24.392050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:43821)
0620 07:17:24.392318 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:24.392332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:24.392424 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:24.392667 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:24.392669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:24.392851 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:24.392958 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:24.392963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:24.393763 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:24.393767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:24.394341 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:24.394350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:24.394827 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:24.394831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:24.394915 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:24.395382 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:24.395394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:24.397370 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
W20260620 07:17:24.397586 20930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:24.392195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:43821 (local address 127.11.123.190:37605)
0620 07:17:24.392368 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:24.392372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:24.392385 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:17:24.392514 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:24.392516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:24.392560 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:24.392628 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:24.393060 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:24.393664 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:24.394468 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:24.394704 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:24.394826 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:24.397483 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:43821: BlockingRecv error: recv got EOF from 127.11.123.131:43821 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 07:17:25 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:25 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:25.029247 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:25.023578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:45407)
0620 07:17:25.023930 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:25.023945 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:25.024040 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:25.024290 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:25.024292 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:25.024474 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:25.024570 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:25.024575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:25.025366 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:25.025369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:25.025948 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:25.025957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:25.026371 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:25.026373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:25.026434 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:25.026906 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:25.026917 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:25.029052 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":129,"threads_started":1}
W20260620 07:17:25.029302 20933 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:25.023635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:45407 (local address 127.11.123.190:37605)
0620 07:17:25.023871 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:25.023875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:25.024013 (+ 138us) server_negotiation.cc:407] Connection header received
0620 07:17:25.024119 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:25.024121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:25.024164 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:25.024240 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:25.024667 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:25.025264 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:25.026058 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:25.026276 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:25.026355 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:25.029183 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:45407: BlockingRecv error: recv got EOF from 127.11.123.129:45407 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
Jun 20 07:17:25 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:25 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:25.233971 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:25.228360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:54475)
0620 07:17:25.228709 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:25.228724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:25.228820 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:25.229069 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:25.229113 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:25.229315 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:25.229412 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:25.229419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:25.230127 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:25.230129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:25.230669 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:25.230676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:25.231090 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:25.231093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:25.231154 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:25.231685 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:25.231736 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:25.233798 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
W20260620 07:17:25.234226 20933 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:25.228454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:54475 (local address 127.11.123.190:37605)
0620 07:17:25.228622 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:25.228629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:25.228799 (+ 170us) server_negotiation.cc:407] Connection header received
0620 07:17:25.228917 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:25.228920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:25.228964 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:25.229031 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:25.229514 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:25.230033 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:25.230767 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:25.230989 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:25.231063 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:25.234116 (+ 3053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:54475: BlockingRecv error: recv got EOF from 127.11.123.130:54475 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 07:17:25 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:25 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:25.403622 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:25.398364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:57399)
0620 07:17:25.398673 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:25.398686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:25.398787 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:25.399030 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:25.399033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:25.399212 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:25.399306 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:25.399311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:25.400082 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:25.400085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:25.400637 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:25.400645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:25.401108 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:25.401110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:25.401174 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:25.401734 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:25.401747 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:25.403513 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":71,"threads_started":1}
W20260620 07:17:25.403802 20933 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:25.398475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:57399 (local address 127.11.123.190:37605)
0620 07:17:25.398725 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:25.398728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:25.398742 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:17:25.398875 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:25.398877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:25.398921 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:25.398989 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:25.399432 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:25.399969 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:25.400762 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:25.400987 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:25.401063 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:25.403642 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:57399: BlockingRecv error: recv got EOF from 127.11.123.131:57399 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 20 07:17:26 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:26 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:26.035262 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:26.030034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:57025)
0620 07:17:26.030301 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:26.030314 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:26.030415 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:26.030643 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:26.030645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:26.030819 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:26.030916 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:26.030921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:26.031665 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:26.031668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:26.032218 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:26.032225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:26.032653 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:26.032655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:26.032718 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:26.033283 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:26.033294 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:26.035133 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
W20260620 07:17:26.035440 20938 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:26.030190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:57025 (local address 127.11.123.190:37605)
0620 07:17:26.030409 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:26.030412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:26.030424 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:17:26.030494 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:26.030496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:26.030537 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:26.030604 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:26.031024 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:26.031569 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:26.032327 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:26.032553 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:26.032633 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:26.035260 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:57025: BlockingRecv error: recv got EOF from 127.11.123.129:57025 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":81,"threads_started":1}
Jun 20 07:17:26 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:26 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:26.240557 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:26.234859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:38719)
0620 07:17:26.235194 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:26.235208 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:26.235308 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:26.235546 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:26.235548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:26.235737 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:26.235833 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:26.235838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:26.236851 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:26.236854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:26.237570 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:26.237581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:26.237999 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:26.238001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:26.238060 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:26.238608 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:26.238629 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:26.240445 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":80,"threads_started":1}
W20260620 07:17:26.240689 20938 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:26.234963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:38719 (local address 127.11.123.190:37605)
0620 07:17:26.235119 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:26.235123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:26.235289 (+ 166us) server_negotiation.cc:407] Connection header received
0620 07:17:26.235393 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:26.235396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:26.235440 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:26.235509 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:26.235996 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:26.236746 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:26.237676 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:26.237896 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:26.237981 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:26.240572 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:38719: BlockingRecv error: recv got EOF from 127.11.123.130:38719 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 07:17:26 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:26 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:26.409349 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:26.404381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:54663)
0620 07:17:26.404686 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:26.404699 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:26.404797 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:26.405024 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:26.405026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:26.405216 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:26.405312 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:26.405317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:26.406064 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:26.406066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:26.406596 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:26.406602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:26.407005 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:26.407008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:26.407070 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:26.407515 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:26.407528 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:26.409238 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":81,"threads_started":1}
W20260620 07:17:26.409495 20938 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:26.404548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:54663 (local address 127.11.123.190:37605)
0620 07:17:26.404695 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:26.404699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:26.404775 (+ 76us) server_negotiation.cc:407] Connection header received
0620 07:17:26.404874 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:26.404876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:26.404920 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:26.404986 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:26.405446 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:26.405972 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:26.406696 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:26.406911 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:26.406988 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:26.409375 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:54663: BlockingRecv error: recv got EOF from 127.11.123.131:54663 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 07:17:27 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:27 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:27.041785 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:27.036248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:36281)
0620 07:17:27.036511 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:27.036526 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:27.036616 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:27.036872 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:27.036874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:27.037055 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:27.037257 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:27.037287 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:27.038057 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:27.038062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:27.038625 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:27.038633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:27.039093 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:27.039097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:27.039184 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:27.039689 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:27.039700 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:27.041602 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":88,"threads_started":1}
W20260620 07:17:27.041883 20942 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:27.036308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:36281 (local address 127.11.123.190:37605)
0620 07:17:27.036543 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:27.036555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:27.036573 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:17:27.036707 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:27.036710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:27.036753 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:27.036830 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:27.037402 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:27.037942 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:27.038752 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:27.038979 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:27.039088 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:27.041717 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:36281: BlockingRecv error: recv got EOF from 127.11.123.129:36281 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":68,"threads_started":1}
Jun 20 07:17:27 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:27 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:27.246388 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:27.241401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:53913)
0620 07:17:27.241691 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:27.241704 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:27.241806 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:27.242019 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:27.242022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:27.242196 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:27.242296 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:27.242301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:27.242995 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:27.242997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:27.243540 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:27.243546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:27.243952 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:27.243954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:27.244013 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:27.244473 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:27.244483 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:27.246274 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":107,"threads_started":1}
W20260620 07:17:27.246520 20942 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:27.241464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:53913 (local address 127.11.123.190:37605)
0620 07:17:27.241597 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:27.241601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:27.241767 (+ 166us) server_negotiation.cc:407] Connection header received
0620 07:17:27.241874 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:27.241876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:27.241917 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:27.241983 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:27.242392 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:27.242908 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:27.243634 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:27.243859 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:27.243933 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:27.246406 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:53913: BlockingRecv error: recv got EOF from 127.11.123.130:53913 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 07:17:27 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:27 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:27.415787 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:27.410182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:57549)
0620 07:17:27.410504 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:27.410516 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:27.410645 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:27.410925 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:27.410928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:27.411132 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:27.411220 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:27.411245 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:27.412026 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:27.412031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:27.412659 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:27.412670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:27.413205 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:27.413211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:27.413290 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:27.413802 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:27.413814 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:27.415656 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":125,"threads_started":1}
W20260620 07:17:27.415872 20942 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:27.410252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:57549 (local address 127.11.123.190:37605)
0620 07:17:27.410438 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:27.410442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:27.410588 (+ 146us) server_negotiation.cc:407] Connection header received
0620 07:17:27.410747 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:27.410750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:27.410807 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:27.410910 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:27.411354 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:27.411914 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:27.412771 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:27.413004 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:27.413160 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:27.415780 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:57549: BlockingRecv error: recv got EOF from 127.11.123.131:57549 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 20 07:17:28 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:28 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:28.047883 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:28.042540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:58033)
0620 07:17:28.042914 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:28.042932 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:28.043031 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:28.043288 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:28.043290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:28.043469 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:28.043568 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:28.043573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:28.044302 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:28.044304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:28.044868 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:28.044875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:28.045339 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:28.045342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:28.045405 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:28.045929 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:28.045940 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:28.047770 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":145,"threads_started":1}
W20260620 07:17:28.048043 20946 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:28.042673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:58033 (local address 127.11.123.190:37605)
0620 07:17:28.042915 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:28.042919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:28.043009 (+ 90us) server_negotiation.cc:407] Connection header received
0620 07:17:28.043111 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:28.043114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:28.043173 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:28.043251 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:28.043676 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:28.044208 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:28.044987 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:28.045238 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:28.045321 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:28.047910 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:58033: BlockingRecv error: recv got EOF from 127.11.123.129:58033 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
Jun 20 07:17:28 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:28 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:28.256487 20947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:28.247285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:46629)
0620 07:17:28.247594 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:28.247607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:28.247748 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:28.248022 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:28.248025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:28.248240 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:28.248334 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:28.248340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:28.249196 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:28.249201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:28.249764 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:28.249772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:28.250241 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:28.250245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:28.250330 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:28.254326 (+ 3996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:28.254338 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:28.256300 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
W20260620 07:17:28.256526 20946 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:28.247348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:46629 (local address 127.11.123.190:37605)
0620 07:17:28.247536 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:28.247541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:28.247688 (+ 147us) server_negotiation.cc:407] Connection header received
0620 07:17:28.247849 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:28.247853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:28.247907 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:28.247995 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:28.248457 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:28.249030 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:28.249894 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:28.250125 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:28.250239 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:28.256419 (+ 6180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:46629: BlockingRecv error: recv got EOF from 127.11.123.130:46629 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 07:17:28 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:28 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:28.422114 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:28.416655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:38989)
0620 07:17:28.416953 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:28.416991 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:28.417162 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:28.417454 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:28.417457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:28.417665 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:28.417753 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:28.417759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:28.418530 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:28.418535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:28.419085 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:28.419092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:28.419559 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:28.419563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:28.419663 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:28.420131 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:28.420142 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:28.421946 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
W20260620 07:17:28.422178 20946 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:28.416839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:38989 (local address 127.11.123.190:37605)
0620 07:17:28.416975 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:28.416980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:28.417093 (+ 113us) server_negotiation.cc:407] Connection header received
0620 07:17:28.417271 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:28.417274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:28.417345 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:28.417434 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:28.417884 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:28.418411 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:28.419205 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:28.419446 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:28.419559 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:28.422072 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:38989: BlockingRecv error: recv got EOF from 127.11.123.131:38989 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 07:17:29 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:29 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:29.054541 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:29.048699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:38091)
0620 07:17:29.048968 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:29.048982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:29.049129 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:29.049372 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:29.049375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:29.049556 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:29.049708 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:29.049714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:29.050521 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:29.050524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:29.051134 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:29.051143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:29.051582 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:29.051593 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:29.051713 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:29.052199 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:29.052211 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:29.054359 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
W20260620 07:17:29.054656 20950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:29.048890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:38091 (local address 127.11.123.190:37605)
0620 07:17:29.049135 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:29.049139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:29.049151 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:17:29.049210 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:29.049212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:29.049253 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:29.049330 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:29.049817 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:29.050411 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:29.051243 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:29.051472 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:29.051551 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:29.054501 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:38091: BlockingRecv error: recv got EOF from 127.11.123.129:38091 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
Jun 20 07:17:29 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:29 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:29.262977 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:29.257355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:38203)
0620 07:17:29.257642 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:29.257655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:29.257757 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:29.257989 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:29.257991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:29.258168 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:29.258272 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:29.258278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:29.259006 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:29.259009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:29.259560 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:29.259568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:29.260115 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:29.260118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:29.260243 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:29.260855 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:29.260872 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:29.262813 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":114,"threads_started":1}
W20260620 07:17:29.263069 20950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:29.257401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:38203 (local address 127.11.123.190:37605)
0620 07:17:29.257561 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:29.257565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:29.257732 (+ 167us) server_negotiation.cc:407] Connection header received
0620 07:17:29.257825 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:29.257827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:29.257883 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:29.257951 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:29.258371 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:29.258888 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:29.259706 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:29.259994 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:29.260084 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:29.262951 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:38203: BlockingRecv error: recv got EOF from 127.11.123.130:38203 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 07:17:29 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:29 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:29.428426 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:29.422925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:34049)
0620 07:17:29.423195 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:29.423209 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:29.423357 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:29.423630 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:29.423633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:29.423844 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:29.423935 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:29.423941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:29.424702 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:29.424707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:29.425372 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:29.425381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:29.425844 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:29.425848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:29.425934 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:29.426481 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:29.426502 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:29.428248 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":86,"threads_started":1}
W20260620 07:17:29.428462 20950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:29.422986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:34049 (local address 127.11.123.190:37605)
0620 07:17:29.423121 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:29.423126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:29.423290 (+ 164us) server_negotiation.cc:407] Connection header received
0620 07:17:29.423465 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:29.423469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:29.423523 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:29.423609 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:29.424053 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:29.424592 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:29.425495 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:29.425722 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:29.425829 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:29.428363 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:34049: BlockingRecv error: recv got EOF from 127.11.123.131:34049 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 07:17:30 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:30 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:30.060629 20953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:30.055353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:49499)
0620 07:17:30.055636 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:30.055649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:30.055756 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:30.055998 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:30.056000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:30.056178 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:30.056291 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:30.056296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:30.057042 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:30.057045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:30.057691 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:30.057701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:30.058117 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:30.058120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:30.058193 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:30.058680 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:30.058691 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:30.060517 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":91,"threads_started":1}
W20260620 07:17:30.060784 20954 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:30.055403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:49499 (local address 127.11.123.190:37605)
0620 07:17:30.055652 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:30.055656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:30.055699 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:17:30.055843 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:30.055845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:30.055886 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:30.055964 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:30.056383 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:30.056915 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:30.057797 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:30.058014 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:30.058088 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:30.060650 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:49499: BlockingRecv error: recv got EOF from 127.11.123.129:49499 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
Jun 20 07:17:30 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:30 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:30.269495 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:30.263896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:42229)
0620 07:17:30.264251 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:30.264267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:30.264407 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:30.264704 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:30.264707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:30.264946 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:30.265066 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:30.265114 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:30.265858 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:30.265864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:30.266429 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:30.266436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:30.266891 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:30.266894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:30.266978 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:30.267552 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:30.267563 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:30.269386 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":85,"threads_started":1}
W20260620 07:17:30.269690 20954 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:30.263982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:42229 (local address 127.11.123.190:37605)
0620 07:17:30.264135 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:30.264140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:30.264370 (+ 230us) server_negotiation.cc:407] Connection header received
0620 07:17:30.264517 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:30.264522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:30.264586 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:30.264708 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:30.265215 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:30.265747 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:30.266549 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:30.266779 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:30.266889 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:30.269525 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:42229: BlockingRecv error: recv got EOF from 127.11.123.130:42229 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 07:17:30 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:30 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:30.434918 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:30.429349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:38073)
0620 07:17:30.429687 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:30.429708 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:30.429837 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:30.430100 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:30.430103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:30.430315 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:30.430404 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:30.430410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:30.431225 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:30.431228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:30.431781 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:30.431788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:30.432252 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:30.432255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:30.432337 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:30.432826 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:30.432837 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:30.434793 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":98,"threads_started":1}
W20260620 07:17:30.435161 20954 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:30.429484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:38073 (local address 127.11.123.190:37605)
0620 07:17:30.429653 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:30.429658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:30.429776 (+ 118us) server_negotiation.cc:407] Connection header received
0620 07:17:30.429933 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:30.429937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:30.429985 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:30.430081 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:30.430516 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:30.431094 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:30.431897 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:30.432142 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:30.432251 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:30.435014 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:38073: BlockingRecv error: recv got EOF from 127.11.123.131:38073 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 07:17:31 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:31 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:31.067232 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:31.061496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:60803)
0620 07:17:31.061819 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:31.061833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:31.061922 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:31.062183 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:31.062185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:31.062375 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:31.062473 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:31.062478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:31.063202 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:31.063204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:31.063960 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:31.063973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:31.064653 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:31.064657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:31.064745 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:31.065284 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:31.065295 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:31.067116 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":117,"threads_started":1}
W20260620 07:17:31.067371 20958 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:31.061627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:60803 (local address 127.11.123.190:37605)
0620 07:17:31.061863 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:31.061867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:31.061881 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:17:31.062014 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:31.062017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:31.062060 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:31.062140 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:31.062576 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:31.063099 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:31.064091 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:31.064493 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:31.064596 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:31.067241 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:60803: BlockingRecv error: recv got EOF from 127.11.123.129:60803 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
Jun 20 07:17:31 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:31 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:31.275557 20959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:31.270294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:49993)
0620 07:17:31.270566 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:31.270579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:31.270662 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:31.270913 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:31.270915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:31.271093 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:31.271187 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:31.271192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:31.271896 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:31.271898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:31.272535 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:31.272547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:31.272990 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:31.272992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:31.273054 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:31.273564 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:31.273575 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:31.275387 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":67,"threads_started":1}
W20260620 07:17:31.275624 20958 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:31.270432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:49993 (local address 127.11.123.190:37605)
0620 07:17:31.270596 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:31.270600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:31.270617 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:17:31.270747 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:31.270750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:31.270792 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:31.270873 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:31.271285 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:31.271793 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:31.272659 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:31.272890 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:31.272969 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:31.275514 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:49993: BlockingRecv error: recv got EOF from 127.11.123.130:49993 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 07:17:31 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:31 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:31.441493 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:31.435722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:59109)
0620 07:17:31.436016 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:31.436029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:31.436168 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:31.436497 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:31.436500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:31.436710 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:31.436846 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:31.436858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:31.437736 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:31.437742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:31.438312 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:31.438320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:31.438774 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:31.438777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:31.438860 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:31.439406 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:31.439420 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:31.441327 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":104,"threads_started":1}
W20260620 07:17:31.441550 20958 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:31.435793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:59109 (local address 127.11.123.190:37605)
0620 07:17:31.435929 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:31.435934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:31.436103 (+ 169us) server_negotiation.cc:407] Connection header received
0620 07:17:31.436327 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:31.436330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:31.436385 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:31.436475 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:31.436963 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:31.437624 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:31.438429 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:31.438659 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:31.438771 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:31.441442 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:59109: BlockingRecv error: recv got EOF from 127.11.123.131:59109 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 07:17:32 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:32 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:32.073145 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:32.067979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:38289)
0620 07:17:32.068285 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:32.068298 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:32.068383 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:32.068620 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:32.068622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:32.068821 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:32.068918 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:32.068923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:32.069694 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:32.069697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:32.070249 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:32.070257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:32.070648 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:32.070650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:32.070726 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:32.071275 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:32.071286 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:32.072974 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":125,"threads_started":1}
W20260620 07:17:32.073294 20962 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:32.068035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:38289 (local address 127.11.123.190:37605)
0620 07:17:32.068320 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:32.068323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:32.068338 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:17:32.068464 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:32.068466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:32.068508 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:32.068583 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:32.069015 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:32.069597 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:32.070343 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:32.070557 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:32.070632 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:32.073130 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:38289: BlockingRecv error: recv got EOF from 127.11.123.129:38289 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
Jun 20 07:17:32 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:32 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:32.281394 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:32.276293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:36265)
0620 07:17:32.276590 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:32.276603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:32.276700 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:32.276909 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:32.276911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:32.277170 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:32.277268 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:32.277274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:32.277986 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:32.277988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:32.278517 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:32.278523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:32.278909 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:32.278911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:32.278969 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:32.279432 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:32.279443 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:32.281272 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":121,"threads_started":1}
W20260620 07:17:32.281514 20962 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:32.276378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:36265 (local address 127.11.123.190:37605)
0620 07:17:32.276525 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:32.276532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:32.276655 (+ 123us) server_negotiation.cc:407] Connection header received
0620 07:17:32.276766 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:32.276768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:32.276810 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:32.276875 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:32.277376 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:32.277898 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:32.278610 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:32.278818 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:32.278890 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:32.281402 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:36265: BlockingRecv error: recv got EOF from 127.11.123.130:36265 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 07:17:32 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:32 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.131@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:32.448133 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:32.442549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.131:45525)
0620 07:17:32.442966 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:32.442987 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:32.443118 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:32.443389 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:32.443392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:32.443603 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:32.443691 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:32.443697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:32.444481 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:32.444484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:32.445059 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:32.445066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:32.445545 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:32.445548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:32.445630 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:32.446149 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:32.446160 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:32.447952 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":106,"threads_started":1}
W20260620 07:17:32.448170 20962 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:32.442583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:45525 (local address 127.11.123.190:37605)
0620 07:17:32.442761 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:32.442766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:32.443067 (+ 301us) server_negotiation.cc:407] Connection header received
0620 07:17:32.443211 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:32.443216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:32.443278 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:32.443366 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:32.443843 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:32.444364 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:32.445204 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:32.445437 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:32.445546 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:32.448065 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.131:45525: BlockingRecv error: recv got EOF from 127.11.123.131:45525 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 07:17:33 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:33 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.129@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:33.079447 20965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:33.074159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.129:59547)
0620 07:17:33.074453 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:33.074470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:33.074568 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:33.074858 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:33.074860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:33.075042 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:33.075130 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:33.075136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.075870 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.075873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:33.076433 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:33.076440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.076891 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.076894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:33.076967 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:33.077527 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:33.077543 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:33.079338 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":74,"threads_started":1}
W20260620 07:17:33.079658 20966 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:33.074256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:59547 (local address 127.11.123.190:37605)
0620 07:17:33.074481 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:33.074484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:33.074539 (+ 55us) server_negotiation.cc:407] Connection header received
0620 07:17:33.074679 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:33.074682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:33.074739 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:33.074812 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:33.075236 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.075780 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.076548 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.076787 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.076869 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:33.079471 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.129:59547: BlockingRecv error: recv got EOF from 127.11.123.129:59547 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
Jun 20 07:17:33 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:17:33 dist-test-slave-j663 krb5kdc[19907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.123.130@KRBTEST.COM for oryx/127.11.123.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:17:33.291680 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:33.282321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37605 (local address 127.11.123.130:59527)
0620 07:17:33.282590 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:33.282603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:33.282697 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:33.282911 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:33.282913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:33.283089 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:33.283181 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:33.283185 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.283877 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.283879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:33.284415 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:33.284421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.284857 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.284859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:33.284923 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:33.289527 (+ 4604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:33.289538 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:33.291555 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.123.190:37605: Server oryx/127.11.123.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
W20260620 07:17:33.291829 20966 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:33.282380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:59527 (local address 127.11.123.190:37605)
0620 07:17:33.282526 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:33.282532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:33.282655 (+ 123us) server_negotiation.cc:407] Connection header received
0620 07:17:33.282771 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:33.282773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:33.282815 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:33.282873 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:33.283277 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.283790 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.284543 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.284757 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.284835 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:33.291678 (+ 6843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.123.130:59527: BlockingRecv error: recv got EOF from 127.11.123.130:59527 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:17:33.347088 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 20490
I20260620 07:17:33.353193 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 20643
I20260620 07:17:33.358706 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 20775
I20260620 07:17:33.364349 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 20420
2026-06-20T07:17:33Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17936 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T07:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:33Z Disabled control of system clock
I20260620 07:17:33.395118 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:41337
--webserver_interface=127.11.123.190
--webserver_port=0
--builtin_ntp_servers=127.11.123.148:39011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:41337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 07:17:33.504235 20971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:33.504482 20971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:33.504513 20971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:33.508499 20971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:33.508558 20971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:33.508574 20971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:33.508589 20971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:33.508651 20971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:17:33.513418 20971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:41337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:41337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:33.514515 20971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:33.515939 20971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:33.524446 20979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:33.524657 20971 server_base.cc:1034] running on GCE node
W20260620 07:17:33.524447 20977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:33.524446 20976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:33.525348 20971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:33.526443 20971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:33.527603 20971 hybrid_clock.cc:648] HybridClock initialized: now 1781939853527590 us; error 25 us; skew 500 ppm
I20260620 07:17:33.529445 20971 webserver.cc:460] Webserver started at http://127.11.123.190:45583/ using document root <none> and password file <none>
I20260620 07:17:33.530014 20971 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:33.530074 20971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:33.530313 20971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:33.532285 20971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "2a459b3a310747a1b923b3954e9df42d"
format_stamp: "Formatted at 2026-06-20 07:17:33 on dist-test-slave-j663"
server_key: "d0e3268f7fb8ae04412ff96e85cda358"
server_key_iv: "cf06109482866c95e39a36b64fd9dc42"
server_key_version: "encryptionkey@0"
I20260620 07:17:33.532845 20971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "2a459b3a310747a1b923b3954e9df42d"
format_stamp: "Formatted at 2026-06-20 07:17:33 on dist-test-slave-j663"
server_key: "d0e3268f7fb8ae04412ff96e85cda358"
server_key_iv: "cf06109482866c95e39a36b64fd9dc42"
server_key_version: "encryptionkey@0"
I20260620 07:17:33.536362 20971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:17:33.538693 20985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:33.539852 20971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:33.539987 20971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "2a459b3a310747a1b923b3954e9df42d"
format_stamp: "Formatted at 2026-06-20 07:17:33 on dist-test-slave-j663"
server_key: "d0e3268f7fb8ae04412ff96e85cda358"
server_key_iv: "cf06109482866c95e39a36b64fd9dc42"
server_key_version: "encryptionkey@0"
I20260620 07:17:33.540096 20971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:33.555248 20971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:33.555881 20971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:33.556075 20971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:33.562994 20971 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:41337
I20260620 07:17:33.563004 21037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:41337 every 8 connection(s)
I20260620 07:17:33.563915 20971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:33.567451 21038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:33.570468 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 20971
I20260620 07:17:33.570593 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:17:33.570904 11758 external_mini_cluster.cc:1428] Setting key fac90ca55592842e6b05d344afe78972
I20260620 07:17:33.574138 21038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d: Bootstrap starting.
I20260620 07:17:33.576584 21038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:33.577417 21038 log.cc:824] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:33.579725 21038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d: No bootstrap required, opened a new log
I20260620 07:17:33.580367 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:33.572307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.11.123.190:41337)
0620 07:17:33.572938 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:33.572953 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:33.572985 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:17:33.573644 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:33.573661 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:33.574035 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:33.574364 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:33.574702 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.575624 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.576799 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.577311 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.577425 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:33.578299 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:33.578308 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:33.578327 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:33.578816 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:33.578887 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:33.578916 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:33.579022 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:33.579139 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 07:17:33.579396 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":136,"threads_started":1}
I20260620 07:17:33.583283 21038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a459b3a310747a1b923b3954e9df42d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 41337 } }
I20260620 07:17:33.583511 21038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:33.583590 21038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a459b3a310747a1b923b3954e9df42d, State: Initialized, Role: FOLLOWER
I20260620 07:17:33.583997 21038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a459b3a310747a1b923b3954e9df42d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 41337 } }
I20260620 07:17:33.584146 21038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:33.584229 21038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:33.584354 21038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:33.585278 21038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a459b3a310747a1b923b3954e9df42d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 41337 } }
I20260620 07:17:33.585613 21038 leader_election.cc:302] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a459b3a310747a1b923b3954e9df42d; no voters:
I20260620 07:17:33.585901 21038 leader_election.cc:288] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:17:33.586037 21043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:33.586251 21043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [term 1 LEADER]: Becoming Leader. State: Replica: 2a459b3a310747a1b923b3954e9df42d, State: Running, Role: LEADER
I20260620 07:17:33.586650 21043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a459b3a310747a1b923b3954e9df42d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 41337 } }
I20260620 07:17:33.587109 21038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:33.588227 21044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a459b3a310747a1b923b3954e9df42d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a459b3a310747a1b923b3954e9df42d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 41337 } } }
I20260620 07:17:33.588371 21044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:33.588762 21052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:33.588784 21045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a459b3a310747a1b923b3954e9df42d. Latest consensus state: current_term: 1 leader_uuid: "2a459b3a310747a1b923b3954e9df42d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a459b3a310747a1b923b3954e9df42d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 41337 } } }
I20260620 07:17:33.588968 21045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:33.591759 21052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:33.597018 21052 catalog_manager.cc:1347] Generated new cluster ID: 56b49fc5c9234317b18397b5c28ae7c3
I20260620 07:17:33.597111 21052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:33.612318 21052 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:17:33.613261 21052 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:33.624761 21052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a459b3a310747a1b923b3954e9df42d: Generated new TSK 0
I20260620 07:17:33.625470 21052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:17:33.630089 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:41337
--builtin_ntp_servers=127.11.123.148:39011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 07:17:33.754076 21062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:33.754320 21062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:33.754381 21062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:33.758441 21062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:33.758490 21062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:33.758540 21062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:17:33.763114 21062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:41337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:33.764119 21062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:33.765550 21062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:33.774655 21070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:33.774638 21068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:33.774631 21067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:33.774879 21062 server_base.cc:1034] running on GCE node
I20260620 07:17:33.775312 21062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:33.775967 21062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:33.777146 21062 hybrid_clock.cc:648] HybridClock initialized: now 1781939853777108 us; error 47 us; skew 500 ppm
I20260620 07:17:33.778810 21062 webserver.cc:460] Webserver started at http://127.11.123.129:45409/ using document root <none> and password file <none>
I20260620 07:17:33.779366 21062 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:33.779456 21062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:33.779675 21062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:33.781672 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "c1dfdd190c63468f9cae53d820cfdf7a"
format_stamp: "Formatted at 2026-06-20 07:17:33 on dist-test-slave-j663"
server_key: "9c4f59fb5f472281d35d9f374d57766a"
server_key_iv: "3bb235471171d75025a0fd3eb29ffcd5"
server_key_version: "encryptionkey@0"
I20260620 07:17:33.782219 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "c1dfdd190c63468f9cae53d820cfdf7a"
format_stamp: "Formatted at 2026-06-20 07:17:33 on dist-test-slave-j663"
server_key: "9c4f59fb5f472281d35d9f374d57766a"
server_key_iv: "3bb235471171d75025a0fd3eb29ffcd5"
server_key_version: "encryptionkey@0"
I20260620 07:17:33.785609 21062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:17:33.788058 21076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:33.789434 21062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:33.789551 21062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "c1dfdd190c63468f9cae53d820cfdf7a"
format_stamp: "Formatted at 2026-06-20 07:17:33 on dist-test-slave-j663"
server_key: "9c4f59fb5f472281d35d9f374d57766a"
server_key_iv: "3bb235471171d75025a0fd3eb29ffcd5"
server_key_version: "encryptionkey@0"
I20260620 07:17:33.789659 21062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:33.799471 21062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:33.800069 21062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:33.800248 21062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:33.800803 21062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:33.801707 21062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:33.801751 21062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:33.801824 21062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:33.801855 21062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:33.810328 21062 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:36511
I20260620 07:17:33.810398 21189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:36511 every 8 connection(s)
I20260620 07:17:33.811188 21062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:17:33.816550 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 21062
I20260620 07:17:33.816643 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:17:33.816924 11758 external_mini_cluster.cc:1428] Setting key b66573d1756d08abf977b51d677d5c40
I20260620 07:17:33.818748 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:41337
--builtin_ntp_servers=127.11.123.148:39011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 07:17:33.820412 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:33.812887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:58325 (local address 127.11.123.190:41337)
0620 07:17:33.813017 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:33.813021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:33.814017 (+ 996us) server_negotiation.cc:407] Connection header received
0620 07:17:33.815409 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:33.815413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:33.815472 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:33.815574 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:33.816434 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.817163 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.818497 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.818740 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.818871 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:33.819927 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:33.819931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:33.819933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:33.819946 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:33.819973 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:33.819981 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:33.820058 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:33.820215 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 07:17:33.820277 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:17:33.821331 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:33.813258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:41337 (local address 127.11.123.129:58325)
0620 07:17:33.813874 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:33.813912 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:33.815176 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:33.815741 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:33.815748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:33.815829 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:33.816279 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:33.816292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.817296 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.817306 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:33.818369 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:33.818378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.818905 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.818909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:33.819064 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:33.819680 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:33.819682 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:33.819719 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:33.819780 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:33.820071 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:33.820082 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:33.820094 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 07:17:33.820216 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 07:17:33.820511 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
I20260620 07:17:33.822635 21190 heartbeater.cc:347] Connected to a master server at 127.11.123.190:41337
I20260620 07:17:33.822897 21190 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:33.823397 21190 heartbeater.cc:511] Master 127.11.123.190:41337 requested a full tablet report, sending...
I20260620 07:17:33.825060 21002 ts_manager.cc:194] Registered new tserver with Master: c1dfdd190c63468f9cae53d820cfdf7a (127.11.123.129:36511)
I20260620 07:17:33.826830 21002 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.123.129:58325
W20260620 07:17:33.926499 21193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:33.926726 21193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:33.926761 21193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:33.930668 21193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:33.930714 21193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:33.930764 21193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:17:33.935204 21193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:41337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:33.936090 21193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:33.937386 21193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:33.946516 21199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:33.946506 21198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:33.946637 21193 server_base.cc:1034] running on GCE node
W20260620 07:17:33.946506 21201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:33.947136 21193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:33.947778 21193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:33.948936 21193 hybrid_clock.cc:648] HybridClock initialized: now 1781939853948925 us; error 28 us; skew 500 ppm
I20260620 07:17:33.950644 21193 webserver.cc:460] Webserver started at http://127.11.123.130:40391/ using document root <none> and password file <none>
I20260620 07:17:33.951189 21193 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:33.951251 21193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:33.951501 21193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:33.953599 21193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "2cc5b4217a414c3c803ad4ca08ae9216"
format_stamp: "Formatted at 2026-06-20 07:17:33 on dist-test-slave-j663"
server_key: "227adad8fb030b1e90ee92600f780668"
server_key_iv: "c85cc704fa4ba42fdf81bf63332fe279"
server_key_version: "encryptionkey@0"
I20260620 07:17:33.954133 21193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "2cc5b4217a414c3c803ad4ca08ae9216"
format_stamp: "Formatted at 2026-06-20 07:17:33 on dist-test-slave-j663"
server_key: "227adad8fb030b1e90ee92600f780668"
server_key_iv: "c85cc704fa4ba42fdf81bf63332fe279"
server_key_version: "encryptionkey@0"
I20260620 07:17:33.957656 21193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:17:33.959987 21207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:33.961133 21193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:33.961251 21193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "2cc5b4217a414c3c803ad4ca08ae9216"
format_stamp: "Formatted at 2026-06-20 07:17:33 on dist-test-slave-j663"
server_key: "227adad8fb030b1e90ee92600f780668"
server_key_iv: "c85cc704fa4ba42fdf81bf63332fe279"
server_key_version: "encryptionkey@0"
I20260620 07:17:33.961359 21193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:33.975277 21193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:33.975891 21193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:33.976078 21193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:33.976626 21193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:33.977598 21193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:33.977645 21193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:33.977713 21193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:33.977751 21193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:33.986204 21193 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:37113
I20260620 07:17:33.986233 21320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:37113 every 8 connection(s)
I20260620 07:17:33.987126 21193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:17:33.995275 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 21193
I20260620 07:17:33.995362 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:17:33.995720 11758 external_mini_cluster.cc:1428] Setting key 0850f0f2d1292134bac4b84a25522c42
I20260620 07:17:33.996102 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:33.988946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:54967 (local address 127.11.123.190:41337)
0620 07:17:33.989139 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:33.989144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:33.990169 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 07:17:33.991413 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:33.991415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:33.991459 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:33.991532 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:33.992328 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.992849 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.994092 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.994374 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.994459 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:33.995535 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:33.995539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:33.995543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:33.995558 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:33.995583 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:33.995594 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:33.995669 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:33.995904 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 07:17:33.995963 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:17:33.997022 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:33.989432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:41337 (local address 127.11.123.130:54967)
0620 07:17:33.990032 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:33.990069 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:33.991246 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:33.991677 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:33.991682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:33.991772 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:33.992211 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:33.992222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.992980 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.992984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:33.993974 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:33.993983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:33.994498 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:33.994501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:33.994629 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:33.995247 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:33.995249 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:33.995285 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:33.995372 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:33.995705 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:33.995712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:33.995716 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:17:33.995896 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 07:17:33.996147 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":111,"threads_started":1}
I20260620 07:17:33.997648 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:41337
--builtin_ntp_servers=127.11.123.148:39011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 07:17:33.999763 21321 heartbeater.cc:347] Connected to a master server at 127.11.123.190:41337
I20260620 07:17:34.000088 21321 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:34.000702 21321 heartbeater.cc:511] Master 127.11.123.190:41337 requested a full tablet report, sending...
I20260620 07:17:34.001753 21002 ts_manager.cc:194] Registered new tserver with Master: 2cc5b4217a414c3c803ad4ca08ae9216 (127.11.123.130:37113)
I20260620 07:17:34.002315 21002 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.123.130:54967
W20260620 07:17:34.104197 21324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:34.104421 21324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:34.104455 21324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:34.108475 21324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:34.108525 21324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:34.108574 21324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:17:34.113195 21324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:41337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:34.114125 21324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:34.115387 21324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:34.125030 21329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:34.125073 21330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:34.125026 21332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:34.125344 21324 server_base.cc:1034] running on GCE node
I20260620 07:17:34.125705 21324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:34.126381 21324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:34.127556 21324 hybrid_clock.cc:648] HybridClock initialized: now 1781939854127523 us; error 44 us; skew 500 ppm
I20260620 07:17:34.129325 21324 webserver.cc:460] Webserver started at http://127.11.123.131:45219/ using document root <none> and password file <none>
I20260620 07:17:34.129874 21324 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:34.129935 21324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:34.130175 21324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:34.132098 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "989076395132470babfee8f898f42103"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "02e5f0ab8dd3ab40354593896844f336"
server_key_iv: "b104532f630dd2d8370bffb9bb21c74f"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.132635 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "989076395132470babfee8f898f42103"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "02e5f0ab8dd3ab40354593896844f336"
server_key_iv: "b104532f630dd2d8370bffb9bb21c74f"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.136053 21324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:17:34.138379 21338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.139540 21324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:34.139657 21324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "989076395132470babfee8f898f42103"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "02e5f0ab8dd3ab40354593896844f336"
server_key_iv: "b104532f630dd2d8370bffb9bb21c74f"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.139755 21324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:34.152511 21324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:34.153162 21324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:34.153337 21324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:34.153903 21324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:34.154758 21324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:34.154812 21324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.154877 21324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:34.154919 21324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.163172 21324 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:38475
I20260620 07:17:34.163201 21451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:38475 every 8 connection(s)
I20260620 07:17:34.164003 21324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:17:34.172928 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:34.165879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:39059 (local address 127.11.123.190:41337)
0620 07:17:34.166014 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:34.166018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:34.167013 (+ 995us) server_negotiation.cc:407] Connection header received
0620 07:17:34.168362 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:34.168366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:34.168426 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:34.168533 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:34.169460 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:34.169993 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:34.171109 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:34.171345 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:34.171458 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:34.172437 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:34.172440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:34.172444 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:34.172456 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:34.172483 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:34.172491 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:34.172568 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:34.172727 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 07:17:34.172787 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:17:34.173760 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 21324
I20260620 07:17:34.173810 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:34.166243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:41337 (local address 127.11.123.131:39059)
0620 07:17:34.166858 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:34.166897 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:34.168152 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:34.168695 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:34.168701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:34.168795 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:34.169311 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:34.169326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:34.170136 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:34.170138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:34.170982 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:34.170990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:34.171495 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:34.171498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:34.171658 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:34.172190 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:34.172191 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:34.172227 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:34.172286 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:34.172569 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:34.172580 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:34.172587 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:17:34.172718 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 07:17:34.172993 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":103,"threads_started":1}
I20260620 07:17:34.173882 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:17:34.174211 11758 external_mini_cluster.cc:1428] Setting key 28cfda81a7f9816a1f6fb9a3426ed91c
I20260620 07:17:34.174980 21452 heartbeater.cc:347] Connected to a master server at 127.11.123.190:41337
I20260620 07:17:34.175352 21452 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:34.175843 21452 heartbeater.cc:511] Master 127.11.123.190:41337 requested a full tablet report, sending...
I20260620 07:17:34.176808 21002 ts_manager.cc:194] Registered new tserver with Master: 989076395132470babfee8f898f42103 (127.11.123.131:38475)
I20260620 07:17:34.177369 21002 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.123.131:39059
I20260620 07:17:34.187726 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 07:17:34.190696 21041 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:34.189762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.11.123.190:41337)
0620 07:17:34.189902 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:34.189906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:34.190038 (+ 132us) server_negotiation.cc:407] Connection header received
0620 07:17:34.190188 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:34.190191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:34.190235 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:34.190338 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:34.190574 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54354: BlockingRecv error: recv got EOF from 127.0.0.1:54354 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:17:34.190821 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 21062
I20260620 07:17:34.196470 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 21193
I20260620 07:17:34.201712 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 21324
I20260620 07:17:34.206617 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 20971
2026-06-20T07:17:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (839 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T07:17:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:34Z Disabled control of system clock
I20260620 07:17:34.234323 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:44575
--webserver_interface=127.11.123.190
--webserver_port=0
--builtin_ntp_servers=127.11.123.148:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 07:17:34.338879 21463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:34.339109 21463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:34.339144 21463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:34.342954 21463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:34.342998 21463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:34.343014 21463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:34.343029 21463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:34.343043 21463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:17:34.347500 21463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:44575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:34.348366 21463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:34.349606 21463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:34.357486 21469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:34.357537 21471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:34.357496 21468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:34.357769 21463 server_base.cc:1034] running on GCE node
I20260620 07:17:34.358260 21463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:34.359258 21463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:34.360455 21463 hybrid_clock.cc:648] HybridClock initialized: now 1781939854360443 us; error 26 us; skew 500 ppm
I20260620 07:17:34.362282 21463 webserver.cc:460] Webserver started at http://127.11.123.190:41793/ using document root <none> and password file <none>
I20260620 07:17:34.362838 21463 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:34.362897 21463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:34.363142 21463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:34.365157 21463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "d5f96b7b6f2f4053970ee38625451fe8"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "d19aadd62d6b6098ec93c4bfdb3c94f4"
server_key_iv: "fe70c49e06872ce46a5d4ba35ca7f9f4"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.365710 21463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "d5f96b7b6f2f4053970ee38625451fe8"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "d19aadd62d6b6098ec93c4bfdb3c94f4"
server_key_iv: "fe70c49e06872ce46a5d4ba35ca7f9f4"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.369231 21463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:17:34.371440 21477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.372594 21463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:34.372740 21463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "d5f96b7b6f2f4053970ee38625451fe8"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "d19aadd62d6b6098ec93c4bfdb3c94f4"
server_key_iv: "fe70c49e06872ce46a5d4ba35ca7f9f4"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.372844 21463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:34.377682 21463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:34.378254 21463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:34.378429 21463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:34.384490 21463 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:44575
I20260620 07:17:34.384526 21529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:44575 every 8 connection(s)
I20260620 07:17:34.385380 21463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:34.388325 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:34.388880 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 21463
I20260620 07:17:34.388999 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:17:34.389333 11758 external_mini_cluster.cc:1428] Setting key fbb087fc07414ab2c6b9ee95f116bede
I20260620 07:17:34.394393 21530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8: Bootstrap starting.
I20260620 07:17:34.394539 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:34.390618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.11.123.190:44575)
0620 07:17:34.391207 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:34.391221 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:34.391253 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:17:34.391910 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:34.391927 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:34.392247 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:34.392582 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:34.393352 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:34.393356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:34.393367 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:34.393377 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:34.393425 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:34.393441 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:34.393584 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:34.393659 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 07:17:34.393881 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260620 07:17:34.396739 21530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:34.397487 21530 log.cc:824] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:34.399163 21530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8: No bootstrap required, opened a new log
I20260620 07:17:34.402767 21530 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f96b7b6f2f4053970ee38625451fe8" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44575 } }
I20260620 07:17:34.402966 21530 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:34.402997 21530 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f96b7b6f2f4053970ee38625451fe8, State: Initialized, Role: FOLLOWER
I20260620 07:17:34.403436 21530 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f96b7b6f2f4053970ee38625451fe8" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44575 } }
I20260620 07:17:34.403553 21530 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:34.403630 21530 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:34.403720 21530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:34.404531 21530 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f96b7b6f2f4053970ee38625451fe8" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44575 } }
I20260620 07:17:34.404855 21530 leader_election.cc:302] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5f96b7b6f2f4053970ee38625451fe8; no voters:
I20260620 07:17:34.405181 21530 leader_election.cc:288] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:17:34.405290 21535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:34.405511 21535 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [term 1 LEADER]: Becoming Leader. State: Replica: d5f96b7b6f2f4053970ee38625451fe8, State: Running, Role: LEADER
I20260620 07:17:34.405857 21535 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f96b7b6f2f4053970ee38625451fe8" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44575 } }
I20260620 07:17:34.406437 21530 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:34.407470 21537 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f96b7b6f2f4053970ee38625451fe8. Latest consensus state: current_term: 1 leader_uuid: "d5f96b7b6f2f4053970ee38625451fe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f96b7b6f2f4053970ee38625451fe8" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44575 } } }
I20260620 07:17:34.407605 21537 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:34.407794 21536 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5f96b7b6f2f4053970ee38625451fe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f96b7b6f2f4053970ee38625451fe8" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44575 } } }
I20260620 07:17:34.407879 21536 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:34.408172 21544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:34.410723 21544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:34.415876 21544 catalog_manager.cc:1347] Generated new cluster ID: 9c556df534184d0e87b0a1c333804f28
I20260620 07:17:34.415946 21544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:34.434665 21544 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:17:34.434991 21544 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:34.442633 21544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5f96b7b6f2f4053970ee38625451fe8: Generated new TSK 0
I20260620 07:17:34.443393 21544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:17:34.447301 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:44575
--builtin_ntp_servers=127.11.123.148:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 07:17:34.557026 21554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:34.557312 21554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:34.557351 21554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:34.561313 21554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:34.561360 21554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:34.561458 21554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:17:34.566072 21554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:34.567265 21554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:34.568643 21554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:34.578755 21559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:34.578796 21562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:34.579012 21554 server_base.cc:1034] running on GCE node
W20260620 07:17:34.578755 21560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:34.579453 21554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:34.580106 21554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:34.581283 21554 hybrid_clock.cc:648] HybridClock initialized: now 1781939854581270 us; error 24 us; skew 500 ppm
I20260620 07:17:34.583120 21554 webserver.cc:460] Webserver started at http://127.11.123.129:37475/ using document root <none> and password file <none>
I20260620 07:17:34.583671 21554 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:34.583772 21554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:34.584003 21554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:34.586030 21554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "8d7861003d9f4df5b98bffbb8470aba4"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "2d10fb8cc4a6885ad82fc6ba0bcd8523"
server_key_iv: "70cb2fd9d0701cc5097897e23f266b4b"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.586588 21554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "8d7861003d9f4df5b98bffbb8470aba4"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "2d10fb8cc4a6885ad82fc6ba0bcd8523"
server_key_iv: "70cb2fd9d0701cc5097897e23f266b4b"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.590142 21554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:17:34.592486 21568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.593688 21554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:34.593832 21554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "8d7861003d9f4df5b98bffbb8470aba4"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "2d10fb8cc4a6885ad82fc6ba0bcd8523"
server_key_iv: "70cb2fd9d0701cc5097897e23f266b4b"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.593945 21554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:34.598205 21554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:34.598788 21554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:34.598969 21554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:34.599488 21554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:34.600373 21554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:34.600445 21554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.600508 21554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:34.600556 21554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.609226 21554 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:40693
I20260620 07:17:34.609268 21681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:40693 every 8 connection(s)
I20260620 07:17:34.610117 21554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:17:34.613432 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 21554
I20260620 07:17:34.613547 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:17:34.613823 11758 external_mini_cluster.cc:1428] Setting key 073ad1a6ee8ca270f205ec9021e7af09
I20260620 07:17:34.615679 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:44575
--builtin_ntp_servers=127.11.123.148:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 07:17:34.616515 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:34.611869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:47117 (local address 127.11.123.190:44575)
0620 07:17:34.612032 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:34.612037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:34.613111 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 07:17:34.614607 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:34.614611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:34.614645 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:34.614756 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:34.615982 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:34.615988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:34.615994 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:34.616013 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:34.616040 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:34.616053 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:34.616158 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:34.616287 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 07:17:34.616333 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:17:34.617461 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:34.612268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:44575 (local address 127.11.123.129:47117)
0620 07:17:34.612937 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:34.612976 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:34.614387 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:34.614907 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:34.614914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:34.615035 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:34.615635 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:34.615637 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:34.615677 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:34.615766 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:34.616149 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:34.616156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:34.616160 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:17:34.616306 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:17:34.616485 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260620 07:17:34.618683 21682 heartbeater.cc:347] Connected to a master server at 127.11.123.190:44575
I20260620 07:17:34.618906 21682 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:34.619289 21682 heartbeater.cc:511] Master 127.11.123.190:44575 requested a full tablet report, sending...
I20260620 07:17:34.621066 21494 ts_manager.cc:194] Registered new tserver with Master: 8d7861003d9f4df5b98bffbb8470aba4 (127.11.123.129:40693)
W20260620 07:17:34.726239 21685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:34.726462 21685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:34.726498 21685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:34.730432 21685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:34.730479 21685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:34.730536 21685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:17:34.735040 21685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:34.735921 21685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:34.737191 21685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:34.746423 21690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:34.746420 21691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:34.746608 21685 server_base.cc:1034] running on GCE node
W20260620 07:17:34.746635 21693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:34.747105 21685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:34.747687 21685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:34.748867 21685 hybrid_clock.cc:648] HybridClock initialized: now 1781939854748855 us; error 27 us; skew 500 ppm
I20260620 07:17:34.750651 21685 webserver.cc:460] Webserver started at http://127.11.123.130:35647/ using document root <none> and password file <none>
I20260620 07:17:34.751194 21685 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:34.751281 21685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:34.751518 21685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:34.753557 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "34e2f911d6ae4400819eaf14dc8603ac"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "30f7e22649ab1688eb60a6b36490e4ba"
server_key_iv: "f34d1130b0c932bcf977fe9d33607c66"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.754138 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "34e2f911d6ae4400819eaf14dc8603ac"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "30f7e22649ab1688eb60a6b36490e4ba"
server_key_iv: "f34d1130b0c932bcf977fe9d33607c66"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.757580 21685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:17:34.759853 21699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.761008 21685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:34.761162 21685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "34e2f911d6ae4400819eaf14dc8603ac"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "30f7e22649ab1688eb60a6b36490e4ba"
server_key_iv: "f34d1130b0c932bcf977fe9d33607c66"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.761281 21685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:34.765565 21685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:34.766150 21685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:34.766335 21685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:34.766845 21685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:34.767724 21685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:34.767791 21685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.767858 21685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:34.767910 21685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.777132 21685 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:34635
I20260620 07:17:34.777215 21812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:34635 every 8 connection(s)
I20260620 07:17:34.778007 21685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:17:34.782119 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 21685
I20260620 07:17:34.782270 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:17:34.782617 11758 external_mini_cluster.cc:1428] Setting key 1addc80c63813ca2c14a8c994ebace90
I20260620 07:17:34.784566 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:34.779808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:44385 (local address 127.11.123.190:44575)
0620 07:17:34.779964 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:34.779969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:34.781011 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 07:17:34.782637 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:34.782640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:34.782727 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:34.782818 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:34.784148 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:34.784151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:34.784153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:34.784163 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:34.784185 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:34.784192 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:34.784260 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:34.784436 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 07:17:34.784470 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:17:34.784776 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:44575
--builtin_ntp_servers=127.11.123.148:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 07:17:34.785785 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:34.780209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:44575 (local address 127.11.123.130:44385)
0620 07:17:34.780858 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:34.780897 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:34.782410 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:34.783027 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:34.783038 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:34.783150 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:34.783859 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:34.783861 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:34.783902 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:34.783968 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:34.784281 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:34.784290 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:34.784297 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:17:34.784476 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 07:17:34.784728 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
I20260620 07:17:34.786947 21813 heartbeater.cc:347] Connected to a master server at 127.11.123.190:44575
I20260620 07:17:34.787174 21813 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:34.787559 21813 heartbeater.cc:511] Master 127.11.123.190:44575 requested a full tablet report, sending...
I20260620 07:17:34.788546 21494 ts_manager.cc:194] Registered new tserver with Master: 34e2f911d6ae4400819eaf14dc8603ac (127.11.123.130:34635)
W20260620 07:17:34.896073 21816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:34.896319 21816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:34.896385 21816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:34.900530 21816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:34.900606 21816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:34.900694 21816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:17:34.905217 21816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:34.906145 21816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:34.907533 21816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:34.917451 21822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:34.917863 21821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:34.918205 21824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:34.921800 21816 server_base.cc:1034] running on GCE node
I20260620 07:17:34.922315 21816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:34.923062 21816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:34.924233 21816 hybrid_clock.cc:648] HybridClock initialized: now 1781939854924167 us; error 79 us; skew 500 ppm
I20260620 07:17:34.926486 21816 webserver.cc:460] Webserver started at http://127.11.123.131:39613/ using document root <none> and password file <none>
I20260620 07:17:34.927178 21816 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:34.927263 21816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:34.927544 21816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:34.930024 21816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "6ae514cf607e466cb73711a04ec35218"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "e265cb114693066ecc39e4c25f01f3a6"
server_key_iv: "d77da4ecc4979a18814a4d6a11ae649d"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.930678 21816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "6ae514cf607e466cb73711a04ec35218"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "e265cb114693066ecc39e4c25f01f3a6"
server_key_iv: "d77da4ecc4979a18814a4d6a11ae649d"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.935760 21816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:17:34.939219 21830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.941006 21816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:17:34.941166 21816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "6ae514cf607e466cb73711a04ec35218"
format_stamp: "Formatted at 2026-06-20 07:17:34 on dist-test-slave-j663"
server_key: "e265cb114693066ecc39e4c25f01f3a6"
server_key_iv: "d77da4ecc4979a18814a4d6a11ae649d"
server_key_version: "encryptionkey@0"
I20260620 07:17:34.941306 21816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:34.948032 21816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:34.948982 21816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:34.949214 21816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:34.949846 21816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:34.950980 21816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:34.951048 21816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.951107 21816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:34.951140 21816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:34.962579 21816 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:46771
I20260620 07:17:34.962615 21943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:46771 every 8 connection(s)
I20260620 07:17:34.963727 21816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:17:34.971788 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:34.966041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:48663 (local address 127.11.123.190:44575)
0620 07:17:34.966192 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:34.966197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:34.967473 (+ 1276us) server_negotiation.cc:407] Connection header received
0620 07:17:34.969531 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:34.969534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:34.969600 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:34.969741 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:34.971212 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:34.971215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:34.971218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:17:34.971229 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:34.971272 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:17:34.971285 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:34.971396 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:34.971570 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 07:17:34.971609 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:17:34.971915 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 21816
I20260620 07:17:34.972008 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:17:34.972292 11758 external_mini_cluster.cc:1428] Setting key c84fe13b6cb92c44e613cee8752bd98c
I20260620 07:17:34.972812 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:34.966471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:44575 (local address 127.11.123.131:48663)
0620 07:17:34.967292 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:34.967351 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:34.969261 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:34.969953 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:34.969966 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:34.970123 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:34.970871 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:17:34.970875 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:34.970933 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:17:34.971022 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:34.971379 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:34.971388 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:34.971395 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:17:34.971579 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 07:17:34.971824 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":118,"threads_started":1}
I20260620 07:17:34.974632 21944 heartbeater.cc:347] Connected to a master server at 127.11.123.190:44575
I20260620 07:17:34.974870 21944 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:34.975353 21944 heartbeater.cc:511] Master 127.11.123.190:44575 requested a full tablet report, sending...
I20260620 07:17:34.976476 21493 ts_manager.cc:194] Registered new tserver with Master: 6ae514cf607e466cb73711a04ec35218 (127.11.123.131:46771)
I20260620 07:17:34.985749 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 07:17:34.988340 21533 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:17:34.987515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.11.123.190:44575)
0620 07:17:34.987645 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:34.987650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:34.987856 (+ 206us) server_negotiation.cc:407] Connection header received
0620 07:17:34.987931 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:34.987934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:34.987960 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:34.988019 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:34.988229 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43496: BlockingRecv error: recv got EOF from 127.0.0.1:43496 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:17:34.988513 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 21554
I20260620 07:17:34.994065 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 21685
I20260620 07:17:34.998878 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 21816
I20260620 07:17:35.003916 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 21463
2026-06-20T07:17:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (797 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:17:35 dist-test-slave-j663 krb5kdc[21954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:17:35 dist-test-slave-j663 krb5kdc[21954](info): set up 2 sockets
Jun 20 07:17:35 dist-test-slave-j663 krb5kdc[21954](info): commencing operation
krb5kdc: starting...
W20260620 07:17:36.041975 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:17:36 dist-test-slave-j663 krb5kdc[21954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:17:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:17:36.157554 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:33695
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:33695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:36.265422 21967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:36.265707 21967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:36.265776 21967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:36.269804 21967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:17:36.269888 21967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:36.269925 21967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:36.269984 21967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:17:36.270025 21967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:17:36.275179 21967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:33695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:33695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:36.276206 21967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:36.277297 21967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:36.285876 21973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:36.285887 21975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:36.285899 21972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:36.286298 21967 server_base.cc:1034] running on GCE node
I20260620 07:17:36.286844 21967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:36.287923 21967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:36.289144 21967 hybrid_clock.cc:648] HybridClock initialized: now 1781939856289123 us; error 29 us; skew 500 ppm
Jun 20 07:17:36 dist-test-slave-j663 krb5kdc[21954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939856, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:36.291867 21967 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:17:36.292743 21967 webserver.cc:460] Webserver started at http://127.11.123.190:32849/ using document root <none> and password file <none>
I20260620 07:17:36.293355 21967 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:36.293442 21967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:36.293675 21967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:36.295599 21967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "57c0d3a466ad4325855c576ee9669702"
format_stamp: "Formatted at 2026-06-20 07:17:36 on dist-test-slave-j663"
server_key: "3b5756db09af9a0dd60192930ae0b42d"
server_key_iv: "1d6128b148201b5de749f901c9f755df"
server_key_version: "encryptionkey@0"
I20260620 07:17:36.296139 21967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "57c0d3a466ad4325855c576ee9669702"
format_stamp: "Formatted at 2026-06-20 07:17:36 on dist-test-slave-j663"
server_key: "3b5756db09af9a0dd60192930ae0b42d"
server_key_iv: "1d6128b148201b5de749f901c9f755df"
server_key_version: "encryptionkey@0"
I20260620 07:17:36.299705 21967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:17:36.301956 21982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:36.303090 21967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:36.303224 21967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "57c0d3a466ad4325855c576ee9669702"
format_stamp: "Formatted at 2026-06-20 07:17:36 on dist-test-slave-j663"
server_key: "3b5756db09af9a0dd60192930ae0b42d"
server_key_iv: "1d6128b148201b5de749f901c9f755df"
server_key_version: "encryptionkey@0"
I20260620 07:17:36.303328 21967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:36.320830 21967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:36.324036 21967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:36.324223 21967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:36.331076 21967 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:33695
I20260620 07:17:36.331346 22034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:33695 every 8 connection(s)
I20260620 07:17:36.332087 21967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:17:36.332847 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 21967
I20260620 07:17:36.332966 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:17:36.333313 11758 external_mini_cluster.cc:1428] Setting key 117d7cf12385b027fc2bb8b920ca9e07
I20260620 07:17:36.335489 22035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:36.341889 22035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702: Bootstrap starting.
Jun 20 07:17:36 dist-test-slave-j663 krb5kdc[21954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:36.344362 22035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:36.345265 22035 log.cc:824] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:36.347429 22035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702: No bootstrap required, opened a new log
I20260620 07:17:36.348240 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:36.334613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.11.123.190:33695)
0620 07:17:36.335007 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:36.335025 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:36.335066 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:17:36.335887 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:36.335909 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:36.336306 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:36.336656 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:36.337385 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:36.338883 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:36.339740 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:36.340422 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:36.340581 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:36.343234 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:36.343239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:36.343255 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:36.343285 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:36.345256 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:36.345723 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:36.345728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:36.345732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:36.345832 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:36.346138 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:36.346143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:36.346145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:36.346610 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:36.346790 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:36.347076 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 07:17:36.347360 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
I20260620 07:17:36.351222 22035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c0d3a466ad4325855c576ee9669702" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33695 } }
I20260620 07:17:36.351414 22035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:36.351446 22035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c0d3a466ad4325855c576ee9669702, State: Initialized, Role: FOLLOWER
I20260620 07:17:36.351836 22035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c0d3a466ad4325855c576ee9669702" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33695 } }
I20260620 07:17:36.351949 22035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:17:36.351994 22035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:17:36.352058 22035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:36.352821 22035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c0d3a466ad4325855c576ee9669702" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33695 } }
I20260620 07:17:36.353155 22035 leader_election.cc:302] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57c0d3a466ad4325855c576ee9669702; no voters:
I20260620 07:17:36.353394 22035 leader_election.cc:288] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:17:36.353525 22040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:36.353793 22040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [term 1 LEADER]: Becoming Leader. State: Replica: 57c0d3a466ad4325855c576ee9669702, State: Running, Role: LEADER
I20260620 07:17:36.354089 22040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c0d3a466ad4325855c576ee9669702" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33695 } }
I20260620 07:17:36.354553 22035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:17:36.355763 22041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57c0d3a466ad4325855c576ee9669702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c0d3a466ad4325855c576ee9669702" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33695 } } }
I20260620 07:17:36.355892 22041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:36.355763 22042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57c0d3a466ad4325855c576ee9669702. Latest consensus state: current_term: 1 leader_uuid: "57c0d3a466ad4325855c576ee9669702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c0d3a466ad4325855c576ee9669702" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33695 } } }
I20260620 07:17:36.356177 22042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702 [sys.catalog]: This master's current role is: LEADER
I20260620 07:17:36.356390 22049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:17:36.359074 22049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:17:36.364274 22049 catalog_manager.cc:1347] Generated new cluster ID: 9831f4f79e574c38b69da67ac4889326
I20260620 07:17:36.364346 22049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:17:36.375079 22049 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:17:36.376307 22049 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:17:36.388906 22049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702: Generated new TSK 0
I20260620 07:17:36.389708 22049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:17:36.419770 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33695
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:36.527154 22061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:36.527376 22061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:36.527416 22061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:36.531325 22061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:36.531373 22061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:36.531463 22061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:17:36.536564 22061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:36.537526 22061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:36.538489 22061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:36.547690 22067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:36.547622 22069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:36.547612 22066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:36.547958 22061 server_base.cc:1034] running on GCE node
I20260620 07:17:36.548384 22061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:36.549222 22061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:36.550426 22061 hybrid_clock.cc:648] HybridClock initialized: now 1781939856550378 us; error 61 us; skew 500 ppm
Jun 20 07:17:36 dist-test-slave-j663 krb5kdc[21954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939856, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:36.553181 22061 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:17:36.553951 22061 webserver.cc:460] Webserver started at http://127.11.123.129:35543/ using document root <none> and password file <none>
I20260620 07:17:36.554443 22061 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:36.554498 22061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:36.554654 22061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:36.556849 22061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "4442cfc6be4b456eb4385a802938d031"
format_stamp: "Formatted at 2026-06-20 07:17:36 on dist-test-slave-j663"
server_key: "42123803f36822383bfefa1f4ede3d9c"
server_key_iv: "0faa14bbb2fd712bacb93e6c6565e3e5"
server_key_version: "encryptionkey@0"
I20260620 07:17:36.557492 22061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "4442cfc6be4b456eb4385a802938d031"
format_stamp: "Formatted at 2026-06-20 07:17:36 on dist-test-slave-j663"
server_key: "42123803f36822383bfefa1f4ede3d9c"
server_key_iv: "0faa14bbb2fd712bacb93e6c6565e3e5"
server_key_version: "encryptionkey@0"
I20260620 07:17:36.560986 22061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:17:36.563338 22076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:36.564508 22061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:17:36.564677 22061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "4442cfc6be4b456eb4385a802938d031"
format_stamp: "Formatted at 2026-06-20 07:17:36 on dist-test-slave-j663"
server_key: "42123803f36822383bfefa1f4ede3d9c"
server_key_iv: "0faa14bbb2fd712bacb93e6c6565e3e5"
server_key_version: "encryptionkey@0"
I20260620 07:17:36.564780 22061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:36.584992 22061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:36.588551 22061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:36.588768 22061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:36.589402 22061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:36.590265 22061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:36.590308 22061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:36.590377 22061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:36.590406 22061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:36.598835 22061 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:40019
I20260620 07:17:36.598922 22189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:40019 every 8 connection(s)
I20260620 07:17:36.599852 22061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:17:36.605436 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 22061
I20260620 07:17:36.605559 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:17:36.605870 11758 external_mini_cluster.cc:1428] Setting key 68381229d942081211d4d03564f417b6
Jun 20 07:17:36 dist-test-slave-j663 krb5kdc[21954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939856, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:36.613672 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:36.601617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:58669 (local address 127.11.123.190:33695)
0620 07:17:36.601782 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:36.601787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:36.602580 (+ 793us) server_negotiation.cc:407] Connection header received
0620 07:17:36.603857 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:36.603859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:36.603904 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:36.603978 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:36.605049 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:36.605760 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:36.607147 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:36.607460 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:36.607571 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:36.610038 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:36.610041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:36.610044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:36.610066 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:36.611478 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:36.612098 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:36.612101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:36.612102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:36.612173 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:36.612559 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:36.612561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:36.612562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:36.612722 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:36.612797 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:36.613451 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 07:17:36.613551 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:17:36.614957 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:36.602011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33695 (local address 127.11.123.129:58669)
0620 07:17:36.602439 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:36.602478 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:36.603673 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:36.604124 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:36.604130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:36.604457 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:36.604921 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:36.604933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:36.605878 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:36.605884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:36.607045 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:36.607056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:36.607634 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:36.607639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:36.607809 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:36.608447 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:36.608462 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:36.609869 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:36.611592 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:36.611601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:36.611618 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:36.611999 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:36.612261 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:36.612267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:36.612271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:36.612465 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:36.612826 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:36.612836 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:36.613200 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 07:17:36.613423 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 07:17:36.613879 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:17:36.616238 22190 heartbeater.cc:347] Connected to a master server at 127.11.123.190:33695
I20260620 07:17:36.616483 22190 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:36.617045 22190 heartbeater.cc:511] Master 127.11.123.190:33695 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
I20260620 07:17:36.618680 21999 ts_manager.cc:194] Registered new tserver with Master: 4442cfc6be4b456eb4385a802938d031 (127.11.123.129:40019)
I20260620 07:17:36.619813 21999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:58669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:17:36.632201 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33695
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:36.737812 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:36.738044 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:36.738083 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:36.742030 22195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:36.742076 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:36.742177 22195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:17:36.747440 22195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:36.748373 22195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:36.749370 22195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:17:36.758719 22200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:36.758787 22201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:36.758751 22203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:36.759048 22195 server_base.cc:1034] running on GCE node
I20260620 07:17:36.759433 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:36.760085 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:36.761266 22195 hybrid_clock.cc:648] HybridClock initialized: now 1781939856761248 us; error 34 us; skew 500 ppm
Jun 20 07:17:36 dist-test-slave-j663 krb5kdc[21954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939856, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:36.763911 22195 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:17:36.764765 22195 webserver.cc:460] Webserver started at http://127.11.123.130:41269/ using document root <none> and password file <none>
I20260620 07:17:36.765409 22195 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:36.765499 22195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:36.765707 22195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:36.767670 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "bb06b8584805407a8a167c4e14be1ac9"
format_stamp: "Formatted at 2026-06-20 07:17:36 on dist-test-slave-j663"
server_key: "05b0c459314a2e0e3c92a286f90e5995"
server_key_iv: "29063899347b4524ad50ce2717e9f923"
server_key_version: "encryptionkey@0"
I20260620 07:17:36.768247 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "bb06b8584805407a8a167c4e14be1ac9"
format_stamp: "Formatted at 2026-06-20 07:17:36 on dist-test-slave-j663"
server_key: "05b0c459314a2e0e3c92a286f90e5995"
server_key_iv: "29063899347b4524ad50ce2717e9f923"
server_key_version: "encryptionkey@0"
I20260620 07:17:36.771878 22195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 07:17:36.774295 22210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:36.775398 22195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:17:36.775529 22195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "bb06b8584805407a8a167c4e14be1ac9"
format_stamp: "Formatted at 2026-06-20 07:17:36 on dist-test-slave-j663"
server_key: "05b0c459314a2e0e3c92a286f90e5995"
server_key_iv: "29063899347b4524ad50ce2717e9f923"
server_key_version: "encryptionkey@0"
I20260620 07:17:36.775637 22195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:36.794656 22195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:36.797832 22195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:36.798025 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:36.798574 22195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:36.799350 22195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:36.799422 22195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:36.799501 22195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:36.799546 22195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:36.808431 22195 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:46029
I20260620 07:17:36.808456 22323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:46029 every 8 connection(s)
I20260620 07:17:36.809305 22195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:17:36.817500 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 22195
I20260620 07:17:36.817633 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:17:36.817941 11758 external_mini_cluster.cc:1428] Setting key 2f9aee731b60042416b888acd32473bf
Jun 20 07:17:36 dist-test-slave-j663 krb5kdc[21954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939856, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:36.822822 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:36.810945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:49411 (local address 127.11.123.190:33695)
0620 07:17:36.811076 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:36.811080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:36.811844 (+ 764us) server_negotiation.cc:407] Connection header received
0620 07:17:36.813280 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:36.813283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:36.813339 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:36.813448 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:36.814540 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:36.815067 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:36.816181 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:36.816405 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:36.816519 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:36.819004 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:36.819008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:36.819012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:36.819040 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:36.820447 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:36.821053 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:36.821055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:36.821057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:36.821181 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:36.821564 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:36.821567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:36.821568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:36.821743 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:36.821816 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:36.822429 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 07:17:36.822563 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:17:36.823436 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:36.811283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33695 (local address 127.11.123.130:49411)
0620 07:17:36.811706 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:36.811747 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:36.812999 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:36.813618 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:36.813625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:36.813953 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:36.814398 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:36.814410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:36.815200 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:36.815203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:36.816053 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:36.816062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:36.816551 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:36.816556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:36.816720 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:36.817306 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:36.817322 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:36.818857 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:36.820566 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:36.820572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:36.820585 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:36.820943 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:36.821279 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:36.821283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:36.821287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:36.821470 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:36.821848 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:36.821854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:36.822170 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 07:17:36.822348 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 07:17:36.822576 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
I20260620 07:17:36.824658 22324 heartbeater.cc:347] Connected to a master server at 127.11.123.190:33695
I20260620 07:17:36.824889 22324 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:36.825389 22324 heartbeater.cc:511] Master 127.11.123.190:33695 requested a full tablet report, sending...
I20260620 07:17:36.826457 21999 ts_manager.cc:194] Registered new tserver with Master: bb06b8584805407a8a167c4e14be1ac9 (127.11.123.130:46029)
I20260620 07:17:36.827270 21999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:49411
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:17:36.844110 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33695
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:17:36.954744 22329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:17:36.955055 22329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:17:36.955129 22329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:17:36.959386 22329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:17:36.959467 22329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:17:36.959574 22329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:17:36.964663 22329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:17:36.965777 22329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:17:36.966743 22329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:17:36.976540 22329 server_base.cc:1034] running on GCE node
W20260620 07:17:36.976444 22334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:36.976549 22337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:17:36.976512 22335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:17:36.977200 22329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:17:36.977865 22329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:17:36.979065 22329 hybrid_clock.cc:648] HybridClock initialized: now 1781939856979043 us; error 34 us; skew 500 ppm
Jun 20 07:17:36 dist-test-slave-j663 krb5kdc[21954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939856, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:17:36.981928 22329 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:17:36.982750 22329 webserver.cc:460] Webserver started at http://127.11.123.131:38399/ using document root <none> and password file <none>
I20260620 07:17:36.983244 22329 fs_manager.cc:359] Metadata directory not provided
I20260620 07:17:36.983302 22329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:17:36.983458 22329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:17:36.985452 22329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "7bc48801b9764fe894ecaa133253c1e7"
format_stamp: "Formatted at 2026-06-20 07:17:36 on dist-test-slave-j663"
server_key: "64ba77d632559ee6c188fa61daf7f73c"
server_key_iv: "f94f05af3c6aab47658e2423ee85031e"
server_key_version: "encryptionkey@0"
I20260620 07:17:36.985944 22329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "7bc48801b9764fe894ecaa133253c1e7"
format_stamp: "Formatted at 2026-06-20 07:17:36 on dist-test-slave-j663"
server_key: "64ba77d632559ee6c188fa61daf7f73c"
server_key_iv: "f94f05af3c6aab47658e2423ee85031e"
server_key_version: "encryptionkey@0"
I20260620 07:17:36.989540 22329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:17:36.992000 22344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:36.993287 22329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:17:36.993405 22329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "7bc48801b9764fe894ecaa133253c1e7"
format_stamp: "Formatted at 2026-06-20 07:17:36 on dist-test-slave-j663"
server_key: "64ba77d632559ee6c188fa61daf7f73c"
server_key_iv: "f94f05af3c6aab47658e2423ee85031e"
server_key_version: "encryptionkey@0"
I20260620 07:17:36.993558 22329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:17:37.015362 22329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:17:37.018607 22329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:17:37.018757 22329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:17:37.019447 22329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:17:37.020469 22329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:17:37.020538 22329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:37.020596 22329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:17:37.020630 22329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:17:37.030421 22329 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:40889
I20260620 07:17:37.030442 22457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:40889 every 8 connection(s)
I20260620 07:17:37.031275 22329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:17:37.039848 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 22329
I20260620 07:17:37.039963 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:17:37.040318 11758 external_mini_cluster.cc:1428] Setting key 4e905dfc187fb4cceba2d04bf0dddd16
Jun 20 07:17:37 dist-test-slave-j663 krb5kdc[21954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939856, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:17:37.045002 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.032942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:41127 (local address 127.11.123.190:33695)
0620 07:17:37.033112 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:37.033116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:37.034284 (+ 1168us) server_negotiation.cc:407] Connection header received
0620 07:17:37.035677 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:37.035680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:37.035727 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:37.035801 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:37.036871 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.037498 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.038582 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.038785 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.038864 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.041477 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:37.041480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:37.041482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:37.041504 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:37.043051 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:37.043553 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:37.043557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:37.043559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:37.043640 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:37.043963 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:37.043967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:37.043969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:37.044165 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:37.044269 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:37.044731 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 07:17:37.044840 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:17:37.045761 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.033498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33695 (local address 127.11.123.131:41127)
0620 07:17:37.034112 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:37.034173 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:37.035474 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:37.035926 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:37.035932 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:37.036270 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:17:37.036722 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:37.036735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.037618 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.037621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:37.038477 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:37.038485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.038911 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.038913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:37.039045 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.039555 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:17:37.039597 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:17:37.041305 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:17:37.043169 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:37.043175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:37.043188 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:37.043432 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:37.043730 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:17:37.043734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:17:37.043736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:17:37.043868 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:17:37.044276 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:17:37.044283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:17:37.044513 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 07:17:37.044691 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 07:17:37.044978 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":169,"threads_started":1}
I20260620 07:17:37.046756 22458 heartbeater.cc:347] Connected to a master server at 127.11.123.190:33695
I20260620 07:17:37.046995 22458 heartbeater.cc:464] Registering TS with master...
I20260620 07:17:37.047458 22458 heartbeater.cc:511] Master 127.11.123.190:33695 requested a full tablet report, sending...
I20260620 07:17:37.048390 21999 ts_manager.cc:194] Registered new tserver with Master: 7bc48801b9764fe894ecaa133253c1e7 (127.11.123.131:40889)
I20260620 07:17:37.048936 21999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:41127
I20260620 07:17:37.053946 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:17:37.062980 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.055860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.11.123.190:33695)
0620 07:17:37.055997 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:37.056001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:37.056170 (+ 169us) server_negotiation.cc:407] Connection header received
0620 07:17:37.056334 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:37.056337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:37.056388 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:37.056474 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:17:37.057178 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.057671 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.058607 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.058832 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.058942 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.060221 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:17:37.060224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:17:37.060227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:17:37.060247 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:17:37.061621 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:37.061969 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:37.061972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:37.061973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:37.062052 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:17:37.062276 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:17:37.062278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:17:37.062279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:17:37.062463 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:17:37.062575 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:37.062721 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 07:17:37.062822 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:17:37.065590 21999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:17:37.068547 21999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:17:37.085131 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.078154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:40889 (local address 127.0.0.1:43190)
0620 07:17:37.078752 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:37.078766 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:37.078854 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:37.079466 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:37.079470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:37.079500 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:37.079714 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:37.079731 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.081375 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.081380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:37.082991 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:37.083003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.084501 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.084516 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:37.084650 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.084691 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 07:17:37.084804 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 07:17:37.084939 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":103,"threads_started":1}
I20260620 07:17:37.085491 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.077125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:46029 (local address 127.0.0.1:56480)
0620 07:17:37.077921 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:37.077969 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:37.078094 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:37.082052 (+ 3958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:37.082055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:37.082073 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:37.082174 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:37.082182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.083109 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.083115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:37.084087 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:37.084098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.085114 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.085118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:37.085202 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.085226 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:17:37.085299 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 07:17:37.085362 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":78,"threads_started":1}
I20260620 07:17:37.085724 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.077302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.11.123.130:46029)
0620 07:17:37.081686 (+ 4384us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:37.081696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:37.081714 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:17:37.081778 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:37.081781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:37.081916 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:37.082066 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:37.082279 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.082982 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.084239 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.084955 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.085116 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.085364 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:37.085413 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 07:17:37.085498 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4284,"thread_start_us":336,"threads_started":1}
I20260620 07:17:37.085973 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.078551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.11.123.131:40889)
0620 07:17:37.078960 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:37.078976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:37.079002 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:17:37.079084 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:37.079090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:37.079311 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:37.079475 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:37.079851 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.081204 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.083145 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.084130 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.084266 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.084430 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:37.085423 (+ 993us) server_negotiation.cc:299] Negotiation successful
0620 07:17:37.085532 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":150,"threads_started":1}
I20260620 07:17:37.088670 22392 tablet_service.cc:1467] Processing CreateTablet for tablet 9c56b1af436c48c99a13abb74dc7f39f (DEFAULT_TABLE table=test-table [id=49199a58af7045128fab44f6ff38ae59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:37.088677 22258 tablet_service.cc:1467] Processing CreateTablet for tablet 9c56b1af436c48c99a13abb74dc7f39f (DEFAULT_TABLE table=test-table [id=49199a58af7045128fab44f6ff38ae59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:37.089622 22392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c56b1af436c48c99a13abb74dc7f39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:37.089705 22258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c56b1af436c48c99a13abb74dc7f39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:37.089730 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.077228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:40019 (local address 127.0.0.1:37842)
0620 07:17:37.078411 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:37.078425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:37.078550 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:37.085694 (+ 7144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:37.085697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:37.085713 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:37.085806 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:37.085811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.087132 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.087135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:37.088236 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:37.088247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.089356 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.089360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:37.089455 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.089480 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:17:37.089529 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 07:17:37.089590 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"thread_start_us":90,"threads_started":1}
I20260620 07:17:37.090377 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.081359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.11.123.129:40019)
0620 07:17:37.085262 (+ 3903us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:37.085279 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:37.085299 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:17:37.085375 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:37.085381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:37.085565 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:37.085719 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:37.085901 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.086982 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.088381 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.089218 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.089993 (+ 775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.090109 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:37.090157 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 07:17:37.090240 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3793,"thread_start_us":118,"threads_started":1}
I20260620 07:17:37.092880 22124 tablet_service.cc:1467] Processing CreateTablet for tablet 9c56b1af436c48c99a13abb74dc7f39f (DEFAULT_TABLE table=test-table [id=49199a58af7045128fab44f6ff38ae59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:17:37.093861 22124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c56b1af436c48c99a13abb74dc7f39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:17:37.095552 22475 tablet_bootstrap.cc:492] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9: Bootstrap starting.
I20260620 07:17:37.096844 22476 tablet_bootstrap.cc:492] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7: Bootstrap starting.
I20260620 07:17:37.098345 22475 tablet_bootstrap.cc:654] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:37.098981 22476 tablet_bootstrap.cc:654] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:37.099220 22475 log.cc:824] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:37.100185 22476 log.cc:824] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:37.100608 22478 tablet_bootstrap.cc:492] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031: Bootstrap starting.
I20260620 07:17:37.101365 22475 tablet_bootstrap.cc:492] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9: No bootstrap required, opened a new log
I20260620 07:17:37.101565 22475 ts_tablet_manager.cc:1397] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:17:37.102654 22476 tablet_bootstrap.cc:492] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7: No bootstrap required, opened a new log
I20260620 07:17:37.102726 22478 tablet_bootstrap.cc:654] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031: Neither blocks nor log segments found. Creating new log.
I20260620 07:17:37.102826 22476 ts_tablet_manager.cc:1397] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:17:37.103698 22478 log.cc:824] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031: Log is configured to *not* fsync() on all Append() calls
I20260620 07:17:37.105244 22478 tablet_bootstrap.cc:492] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031: No bootstrap required, opened a new log
I20260620 07:17:37.105432 22478 ts_tablet_manager.cc:1397] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:17:37.105368 22475 raft_consensus.cc:348] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb06b8584805407a8a167c4e14be1ac9" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46029 } } peers { permanent_uuid: "4442cfc6be4b456eb4385a802938d031" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 40019 } } peers { permanent_uuid: "7bc48801b9764fe894ecaa133253c1e7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40889 } }
I20260620 07:17:37.105646 22475 raft_consensus.cc:374] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:37.105700 22475 raft_consensus.cc:729] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb06b8584805407a8a167c4e14be1ac9, State: Initialized, Role: FOLLOWER
I20260620 07:17:37.106154 22475 consensus_queue.cc:260] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb06b8584805407a8a167c4e14be1ac9" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46029 } } peers { permanent_uuid: "4442cfc6be4b456eb4385a802938d031" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 40019 } } peers { permanent_uuid: "7bc48801b9764fe894ecaa133253c1e7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40889 } }
I20260620 07:17:37.106628 22476 raft_consensus.cc:348] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb06b8584805407a8a167c4e14be1ac9" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46029 } } peers { permanent_uuid: "4442cfc6be4b456eb4385a802938d031" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 40019 } } peers { permanent_uuid: "7bc48801b9764fe894ecaa133253c1e7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40889 } }
I20260620 07:17:37.106866 22476 raft_consensus.cc:374] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:37.106873 22324 heartbeater.cc:503] Master 127.11.123.190:33695 was elected leader, sending a full tablet report...
I20260620 07:17:37.106912 22476 raft_consensus.cc:729] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc48801b9764fe894ecaa133253c1e7, State: Initialized, Role: FOLLOWER
I20260620 07:17:37.107149 22475 ts_tablet_manager.cc:1428] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:17:37.107311 22476 consensus_queue.cc:260] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb06b8584805407a8a167c4e14be1ac9" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46029 } } peers { permanent_uuid: "4442cfc6be4b456eb4385a802938d031" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 40019 } } peers { permanent_uuid: "7bc48801b9764fe894ecaa133253c1e7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40889 } }
I20260620 07:17:37.107878 22458 heartbeater.cc:503] Master 127.11.123.190:33695 was elected leader, sending a full tablet report...
I20260620 07:17:37.108239 22476 ts_tablet_manager.cc:1428] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 07:17:37.109162 22478 raft_consensus.cc:348] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb06b8584805407a8a167c4e14be1ac9" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46029 } } peers { permanent_uuid: "4442cfc6be4b456eb4385a802938d031" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 40019 } } peers { permanent_uuid: "7bc48801b9764fe894ecaa133253c1e7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40889 } }
I20260620 07:17:37.109388 22478 raft_consensus.cc:374] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:17:37.109488 22478 raft_consensus.cc:729] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4442cfc6be4b456eb4385a802938d031, State: Initialized, Role: FOLLOWER
I20260620 07:17:37.109895 22478 consensus_queue.cc:260] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb06b8584805407a8a167c4e14be1ac9" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46029 } } peers { permanent_uuid: "4442cfc6be4b456eb4385a802938d031" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 40019 } } peers { permanent_uuid: "7bc48801b9764fe894ecaa133253c1e7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40889 } }
I20260620 07:17:37.110544 22190 heartbeater.cc:503] Master 127.11.123.190:33695 was elected leader, sending a full tablet report...
I20260620 07:17:37.110838 22478 ts_tablet_manager.cc:1428] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:17:37.113956 22481 raft_consensus.cc:482] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:17:37.114141 22481 raft_consensus.cc:504] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb06b8584805407a8a167c4e14be1ac9" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46029 } } peers { permanent_uuid: "4442cfc6be4b456eb4385a802938d031" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 40019 } } peers { permanent_uuid: "7bc48801b9764fe894ecaa133253c1e7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40889 } }
I20260620 07:17:37.115139 22481 leader_election.cc:288] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4442cfc6be4b456eb4385a802938d031 (127.11.123.129:40019), 7bc48801b9764fe894ecaa133253c1e7 (127.11.123.131:40889)
I20260620 07:17:37.119325 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.115388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:44557 (local address 127.11.123.129:40019)
0620 07:17:37.115529 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:37.115534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:37.115603 (+ 69us) server_negotiation.cc:407] Connection header received
0620 07:17:37.115886 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:37.115889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:37.115939 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:37.116025 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:37.116346 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.116852 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.118324 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.118723 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.118847 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.118901 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:37.119134 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 07:17:37.119203 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:17:37.119335 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.115750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:48533 (local address 127.11.123.131:40889)
0620 07:17:37.115857 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:37.115863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:37.115878 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:17:37.115915 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:37.115917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:37.115951 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:37.116041 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:17:37.116374 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.116854 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.118320 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.118726 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.118851 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.118906 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:37.119134 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 07:17:37.119207 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:17:37.119356 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.115300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:40019 (local address 127.11.123.130:44557)
0620 07:17:37.115502 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:37.115535 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:37.115631 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:37.116038 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:37.116041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:37.116056 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:37.116233 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:37.116238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.116961 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.116966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:37.118206 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:37.118214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.118837 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.118840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:37.118972 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.119030 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 07:17:37.119119 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 07:17:37.119200 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":161,"spinlock_wait_cycles":24320}
I20260620 07:17:37.119737 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.115377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:40889 (local address 127.11.123.130:48533)
0620 07:17:37.115673 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:37.115688 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:17:37.115767 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:17:37.116053 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:17:37.116056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:17:37.116094 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:17:37.116244 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:37.116249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.116961 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.116966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:37.118203 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:17:37.118212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.118844 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.118847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:17:37.119006 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.119033 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:17:37.119118 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:17:37.119201 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":54,"spinlock_wait_cycles":1792,"thread_start_us":112,"threads_started":1}
I20260620 07:17:37.119971 22144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c56b1af436c48c99a13abb74dc7f39f" candidate_uuid: "bb06b8584805407a8a167c4e14be1ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4442cfc6be4b456eb4385a802938d031" is_pre_election: true
I20260620 07:17:37.120261 22144 raft_consensus.cc:2454] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb06b8584805407a8a167c4e14be1ac9 in term 0.
I20260620 07:17:37.120311 22412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c56b1af436c48c99a13abb74dc7f39f" candidate_uuid: "bb06b8584805407a8a167c4e14be1ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc48801b9764fe894ecaa133253c1e7" is_pre_election: true
I20260620 07:17:37.120575 22412 raft_consensus.cc:2454] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb06b8584805407a8a167c4e14be1ac9 in term 0.
I20260620 07:17:37.120697 22214 leader_election.cc:302] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4442cfc6be4b456eb4385a802938d031, bb06b8584805407a8a167c4e14be1ac9; no voters:
I20260620 07:17:37.120939 22481 raft_consensus.cc:2781] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:17:37.121042 22481 raft_consensus.cc:482] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:17:37.121125 22481 raft_consensus.cc:3037] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:37.122088 22481 raft_consensus.cc:504] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb06b8584805407a8a167c4e14be1ac9" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46029 } } peers { permanent_uuid: "4442cfc6be4b456eb4385a802938d031" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 40019 } } peers { permanent_uuid: "7bc48801b9764fe894ecaa133253c1e7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40889 } }
I20260620 07:17:37.122491 22481 leader_election.cc:288] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [CANDIDATE]: Term 1 election: Requested vote from peers 4442cfc6be4b456eb4385a802938d031 (127.11.123.129:40019), 7bc48801b9764fe894ecaa133253c1e7 (127.11.123.131:40889)
I20260620 07:17:37.122821 22412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c56b1af436c48c99a13abb74dc7f39f" candidate_uuid: "bb06b8584805407a8a167c4e14be1ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc48801b9764fe894ecaa133253c1e7"
I20260620 07:17:37.122848 22144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c56b1af436c48c99a13abb74dc7f39f" candidate_uuid: "bb06b8584805407a8a167c4e14be1ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4442cfc6be4b456eb4385a802938d031"
I20260620 07:17:37.122942 22412 raft_consensus.cc:3037] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:37.122964 22144 raft_consensus.cc:3037] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:17:37.124182 22144 raft_consensus.cc:2454] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb06b8584805407a8a167c4e14be1ac9 in term 1.
I20260620 07:17:37.124182 22412 raft_consensus.cc:2454] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb06b8584805407a8a167c4e14be1ac9 in term 1.
I20260620 07:17:37.124461 22214 leader_election.cc:302] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4442cfc6be4b456eb4385a802938d031, bb06b8584805407a8a167c4e14be1ac9; no voters:
I20260620 07:17:37.124714 22481 raft_consensus.cc:2781] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:17:37.124933 22481 raft_consensus.cc:686] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 1 LEADER]: Becoming Leader. State: Replica: bb06b8584805407a8a167c4e14be1ac9, State: Running, Role: LEADER
I20260620 07:17:37.125288 22481 consensus_queue.cc:237] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb06b8584805407a8a167c4e14be1ac9" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46029 } } peers { permanent_uuid: "4442cfc6be4b456eb4385a802938d031" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 40019 } } peers { permanent_uuid: "7bc48801b9764fe894ecaa133253c1e7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40889 } }
I20260620 07:17:37.128330 21999 catalog_manager.cc:5495] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb06b8584805407a8a167c4e14be1ac9 (127.11.123.130). New cstate: current_term: 1 leader_uuid: "bb06b8584805407a8a167c4e14be1ac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb06b8584805407a8a167c4e14be1ac9" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4442cfc6be4b456eb4385a802938d031" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 40019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bc48801b9764fe894ecaa133253c1e7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260620 07:17:37.149147 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.146020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.11.123.130:46029)
0620 07:17:37.146142 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:17:37.146145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:17:37.146157 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:17:37.146226 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:17:37.146228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:17:37.146279 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:17:37.146367 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:17:37.146588 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.147078 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.147861 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:17:37.148061 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:17:37.148136 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:17:37.148382 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:17:37.148785 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:17:37.148875 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:17:37.148933 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 07:17:37.148976 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:17:37.154995 22144 raft_consensus.cc:1261] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031 [term 1 FOLLOWER]: Refusing update from remote peer bb06b8584805407a8a167c4e14be1ac9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:37.154994 22412 raft_consensus.cc:1261] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7 [term 1 FOLLOWER]: Refusing update from remote peer bb06b8584805407a8a167c4e14be1ac9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:17:37.155813 22485 consensus_queue.cc:1035] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc48801b9764fe894ecaa133253c1e7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:37.155984 22481 consensus_queue.cc:1035] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4442cfc6be4b456eb4385a802938d031" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 40019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:17:37.164870 22489 mvcc.cc:205] Tried to move back new op lower bound from 7298825654899154944 to 7298825654787559424. Current Snapshot: MvccSnapshot[applied={T|T < 7298825654899154944}]
I20260620 07:17:37.171695 21998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:37.171890 21998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:17:37.174453 21998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57c0d3a466ad4325855c576ee9669702: Sending DeleteTablet for 3 replicas of tablet 9c56b1af436c48c99a13abb74dc7f39f
I20260620 07:17:37.175195 22258 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c56b1af436c48c99a13abb74dc7f39f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:37 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:56480
I20260620 07:17:37.175607 22392 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c56b1af436c48c99a13abb74dc7f39f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:37 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:43190
I20260620 07:17:37.175700 22124 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c56b1af436c48c99a13abb74dc7f39f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:17:37 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:37842
I20260620 07:17:37.176092 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 22061
I20260620 07:17:37.176115 22499 tablet_replica.cc:330] T 9c56b1af436c48c99a13abb74dc7f39f P 4442cfc6be4b456eb4385a802938d031: stopping tablet replica
I20260620 07:17:37.176872 22497 tablet_replica.cc:330] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9: stopping tablet replica
I20260620 07:17:37.177237 22497 raft_consensus.cc:2229] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:17:37.177651 22497 raft_consensus.cc:2258] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:37.178036 22498 tablet_replica.cc:330] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7: stopping tablet replica
I20260620 07:17:37.178335 22498 raft_consensus.cc:2229] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:17:37.178639 22498 raft_consensus.cc:2258] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:17:37.180279 22498 ts_tablet_manager.cc:1905] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:37.182760 22497 ts_tablet_manager.cc:1905] T 9c56b1af436c48c99a13abb74dc7f39f P bb06b8584805407a8a167c4e14be1ac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:17:37.183677 22498 ts_tablet_manager.cc:1918] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:17:37.183786 22498 log.cc:1196] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/9c56b1af436c48c99a13abb74dc7f39f
I20260620 07:17:37.184149 22498 ts_tablet_manager.cc:1939] T 9c56b1af436c48c99a13abb74dc7f39f P 7bc48801b9764fe894ecaa133253c1e7: Deleting consensus metadata
W20260620 07:17:37.184260 21986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:40019 (error 108)
I20260620 07:17:37.185099 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 22195
I20260620 07:17:37.186805 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.186444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:40019 (local address 127.0.0.1:37858)
0620 07:17:37.186613 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:37.186695 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:40019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 07:17:37.186990 21986 catalog_manager.cc:4584] TS 4442cfc6be4b456eb4385a802938d031 (127.11.123.129:40019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c56b1af436c48c99a13abb74dc7f39f: Network error: Client connection negotiation failed: client connection to 127.11.123.129:40019: connect: Connection refused (error 111)
I20260620 07:17:37.189487 21984 catalog_manager.cc:4841] TS 7bc48801b9764fe894ecaa133253c1e7 (127.11.123.131:40889): tablet 9c56b1af436c48c99a13abb74dc7f39f (table test-table [id=49199a58af7045128fab44f6ff38ae59]) successfully deleted
I20260620 07:17:37.190433 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 22329
I20260620 07:17:37.190857 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:17:37.190491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:46029 (local address 127.0.0.1:56508)
0620 07:17:37.190715 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 07:17:37.190770 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.130:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260620 07:17:37.196635 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 21967
2026-06-20T07:17:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2194 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:17:37 dist-test-slave-j663 krb5kdc[22503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:17:37 dist-test-slave-j663 krb5kdc[22503](info): set up 2 sockets
Jun 20 07:17:37 dist-test-slave-j663 krb5kdc[22503](info): commencing operation
krb5kdc: starting...
W20260620 07:17:38.242038 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:17:38 dist-test-slave-j663 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:17:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:17:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.123.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.123.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/rangeradmin_127.11.123.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/rangeradmin_127.11.123.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.123.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.123.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/rangerlookup_127.11.123.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/rangerlookup_127.11.123.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.123.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.123.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/HTTP_127.11.123.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/HTTP_127.11.123.148@KRBTEST.COM.keytab.
I20260620 07:17:38.407104 11758 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbzqqUD/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/postgres -l logfile start
2026-06-20 07:17:40.812 UTC [22533] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 07:17:40.812 UTC [22533] LOG: listening on IPv4 address "127.11.123.148", port 47987
2026-06-20 07:17:40.815 UTC [22533] LOG: listening on Unix socket "/tmp/.s.PGSQL.47987"
2026-06-20 07:17:40.850 UTC [22536] LOG: database system was shut down at 2026-06-20 07:17:39 UTC
2026-06-20 07:17:40.854 UTC [22533] LOG: database system is ready to accept connections
I20260620 07:17:41.805917 11758 mini_postgres.cc:96] Postgres bound to 47987
2026-06-20 07:17:41.812 UTC [22544] FATAL: database "slave" does not exist
127.11.123.148:47987 - accepting connections
I20260620 07:17:41.812896 11758 mini_ranger.cc:161] Starting Ranger...
I20260620 07:17:41.828272 11758 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 07:17:42.288223 11758 mini_ranger.cc:87] Created ranger Postgres database
I20260620 07:17:42.288323 11758 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-admin
2026-06-20 07:17:42,588 [I] DB FLAVOR :POSTGRES
2026-06-20 07:17:42,588 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:17:42,588 [I] Checking connection..
2026-06-20 07:17:42,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 07:17:42,853 [I] Checking connection passed.
2026-06-20 07:17:42,853 [I] --------- Verifying version history table ---------
2026-06-20 07:17:42,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:17:43,098 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:17:43,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:17:43,337 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:17:43,337 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 07:17:43,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 07:17:43.559 UTC [22608] WARNING: there is no transaction in progress
2026-06-20 07:17:43,565 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 07:17:43,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:17:43,788 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 07:17:43,789 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:17:43,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 07:17:44,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 07:17:44,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j663.c.gcp-upstream.internal','N') ;"
2026-06-20 07:17:44,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:17:44,739 [I] Table x_portal_user does not exist in database ranger
2026-06-20 07:17:44,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:17:45,006 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 07:17:45,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:17:45,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:17:45,498 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 07:17:45,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 07:17:46.421 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.437 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.453 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.468 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.486 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.505 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.561 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.567 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.581 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.591 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.601 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.615 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.624 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.630 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.636 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:46.642 UTC [22716] WARNING: there is no transaction in progress
2026-06-20 07:17:47,247 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 07:17:47,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 07:17:47,485 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 07:17:47,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:17:47,741 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 07:17:47,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:17:47,993 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 07:17:47,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:17:48,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:17:48,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j663.c.gcp-upstream.internal';"
2026-06-20 07:17:48,658 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 07:17:48,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 07:17:48,905 [I] DB_PATCHES have already been applied
I20260620 07:17:48.911638 11758 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbzqqUD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 07:17:48.911700 11758 mini_ranger.cc:193] Using host: 127.11.123.148
I20260620 07:17:48.915086 11758 mini_ranger.cc:227] Ranger admin URL: http://127.11.123.148:42831
Jun 20, 2026 7:17:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:17:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:17:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:17:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:17:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:17:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:17:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 07:17:50.008893 11758 mini_ranger.cc:160] Time spent starting Ranger: real 8.196s user 0.009s sys 0.001s
Jun 20, 2026 7:18:08 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:18:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 7:18:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:18:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:18:08 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:18:09 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 07:18:10.018563 11758 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.123.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.123.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/rangerkms_127.11.123.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/rangerkms_127.11.123.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.123.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.123.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.123.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/HTTP_127.11.123.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.123.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/HTTP_127.11.123.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 07:18:10 dist-test-slave-j663 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939890, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 07:18:10.099386 11758 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 07:18:10.109882 11758 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 07:18:10.991067 11758 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 07:18:10.991170 11758 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-kms
I20260620 07:18:10.991272 11758 mini_ranger_kms.cc:227] Using postgres at 127.11.123.148:47987
2026-06-20 07:18:11,176 [I] DB FLAVOR :POSTGRES
2026-06-20 07:18:11,176 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:18:11,177 [I] Checking connection
2026-06-20 07:18:11,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 07:18:11,413 [I] connection success
2026-06-20 07:18:11,414 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 07:18:11,414 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 07:18:11,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 07:18:11,641 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 07:18:11,641 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:18:11,642 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 07:18:11,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47987/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 07:18:11,907 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 07:18:12.349745 11758 mini_ranger_kms.cc:310] Created kms service
I20260620 07:18:12.509929 11758 mini_ranger_kms.cc:326] Created kudu user
I20260620 07:18:12.553712 11758 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 07:18:12 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939890, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:18:12.948017 11758 mini_ranger_kms.cc:384] Added ranger policy
I20260620 07:18:12.948175 11758 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 07:18:12.948199 11758 mini_ranger_kms.cc:240] Using host: 127.11.123.148
I20260620 07:18:12.951727 11758 mini_ranger_kms.cc:276] Ranger KMS PID: 22896
I20260620 07:18:12.951853 11758 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.123.148:54973
Jun 20, 2026 7:18:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:18:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:18:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:18:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:18:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:18:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 07:18:15.230505 11758 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.131s user 0.008s sys 0.161s
I20260620 07:18:15.230636 11758 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 07:18:15.230741 11758 mini_ranger_kms.cc:401] 127.11.123.148:54973/kms/v1/keys
Jun 20 07:18:15 dist-test-slave-j663 krb5kdc[22503](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939895, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.123.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 07:18:15 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939895, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.123.148@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
Jun 20 07:18:16 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939895, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.123.148@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
Jun 20 07:18:18 dist-test-slave-j663 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:18:18.451490 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:33515
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:33515
--ranger_config_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.123.148:54973/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:18:18.561575 22947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:18:18.561798 22947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:18:18.561836 22947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:18:18.565800 22947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:18:18.565851 22947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:18:18.565872 22947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:18:18.565973 22947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:18:18.566012 22947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:18:18.571341 22947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.123.148:54973/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:33515
--ranger_config_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:33515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:18:18.572309 22947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:18:18.573390 22947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:18:18.581974 22952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:18:18.581993 22953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:18:18.581988 22955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:18:18.582034 22947 server_base.cc:1034] running on GCE node
I20260620 07:18:18.582675 22947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:18:18.583736 22947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:18:18.584930 22947 hybrid_clock.cc:648] HybridClock initialized: now 1781939898584905 us; error 40 us; skew 500 ppm
Jun 20 07:18:18 dist-test-slave-j663 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939898, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:18:18.587716 22947 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:18:18.588470 22947 webserver.cc:460] Webserver started at http://127.11.123.190:34601/ using document root <none> and password file <none>
I20260620 07:18:18.588943 22947 fs_manager.cc:359] Metadata directory not provided
I20260620 07:18:18.588991 22947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:18:18.589244 22947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:18:18 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939898, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:18:18.636854 22947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "0109103522d14b599215204bb8cf4876"
format_stamp: "Formatted at 2026-06-20 07:18:18 on dist-test-slave-j663"
server_key: "be972aac80ecc7b306521573efb57b83"
server_key_iv: "65796e11e946c5393e5c9fbe3945a9c5"
server_key_version: "kuduclusterkey@0"
I20260620 07:18:18.637622 22947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "0109103522d14b599215204bb8cf4876"
format_stamp: "Formatted at 2026-06-20 07:18:18 on dist-test-slave-j663"
server_key: "be972aac80ecc7b306521573efb57b83"
server_key_iv: "65796e11e946c5393e5c9fbe3945a9c5"
server_key_version: "kuduclusterkey@0"
I20260620 07:18:18.642541 22947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:18:18.660014 22962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:18:18.661918 22947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:18:18.662060 22947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "0109103522d14b599215204bb8cf4876"
format_stamp: "Formatted at 2026-06-20 07:18:18 on dist-test-slave-j663"
server_key: "be972aac80ecc7b306521573efb57b83"
server_key_iv: "65796e11e946c5393e5c9fbe3945a9c5"
server_key_version: "kuduclusterkey@0"
I20260620 07:18:18.662210 22947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:18:18.676028 22947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:18:18.679003 22947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:18:18.679210 22947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:18:18.687147 22947 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:33515
I20260620 07:18:18.687147 23014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:33515 every 8 connection(s)
I20260620 07:18:18.688187 22947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:18:18.691082 23015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:18:18.696914 23015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876: Bootstrap starting.
I20260620 07:18:18.698271 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 22947
I20260620 07:18:18.698405 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:18:18.698697 11758 external_mini_cluster.cc:1428] Setting key 94bd0086aac6ed992c783f59c59f51a9
I20260620 07:18:18.699870 23015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876: Neither blocks nor log segments found. Creating new log.
I20260620 07:18:18.700707 23015 log.cc:824] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876: Log is configured to *not* fsync() on all Append() calls
I20260620 07:18:18.702852 23015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876: No bootstrap required, opened a new log
I20260620 07:18:18.706756 23015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0109103522d14b599215204bb8cf4876" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33515 } }
I20260620 07:18:18.707036 23015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:18:18.707096 23015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0109103522d14b599215204bb8cf4876, State: Initialized, Role: FOLLOWER
I20260620 07:18:18.707631 23015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0109103522d14b599215204bb8cf4876" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33515 } }
I20260620 07:18:18.707813 23015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:18:18.707919 23015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:18:18.708040 23015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:18:18.709129 23015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0109103522d14b599215204bb8cf4876" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33515 } }
Jun 20 07:18:18 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:18:18.709529 23015 leader_election.cc:302] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0109103522d14b599215204bb8cf4876; no voters:
I20260620 07:18:18.709930 23015 leader_election.cc:288] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:18:18.710000 23020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:18:18.710254 23020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [term 1 LEADER]: Becoming Leader. State: Replica: 0109103522d14b599215204bb8cf4876, State: Running, Role: LEADER
I20260620 07:18:18.710567 23020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0109103522d14b599215204bb8cf4876" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33515 } }
I20260620 07:18:18.711252 23015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:18:18.711746 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0109103522d14b599215204bb8cf4876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0109103522d14b599215204bb8cf4876" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33515 } } }
I20260620 07:18:18.711887 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [sys.catalog]: This master's current role is: LEADER
I20260620 07:18:18.712121 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0109103522d14b599215204bb8cf4876. Latest consensus state: current_term: 1 leader_uuid: "0109103522d14b599215204bb8cf4876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0109103522d14b599215204bb8cf4876" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33515 } } }
I20260620 07:18:18.712257 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876 [sys.catalog]: This master's current role is: LEADER
I20260620 07:18:18.713006 23025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:18:18.713528 23015 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 07:18:18.715839 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:18.699873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.11.123.190:33515)
0620 07:18:18.700374 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 07:18:18.700393 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:18:18.700437 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:18:18.701297 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:18:18.701321 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:18:18.701731 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:18:18.702103 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:18:18.702986 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:18.704530 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:18.705826 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:18.706566 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:18.706733 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:18.709933 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:18:18.709940 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:18:18.709957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:18:18.709988 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:18:18.712368 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:18:18.712865 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:18:18.712871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:18:18.712876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:18:18.713017 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:18:18.713406 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:18:18.713409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:18:18.713411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:18:18.713971 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:18:18.714173 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 07:18:18.714463 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 07:18:18.714903 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":165,"threads_started":1}
I20260620 07:18:18.717206 23025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:18:18.728320 23025 catalog_manager.cc:1347] Generated new cluster ID: 168805f43d62456da56ef3df70f9f645
I20260620 07:18:18.728399 23025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:18:18.743655 23025 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:18:18.744968 23025 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:18:18.751017 23025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876: Generated new TSK 0
I20260620 07:18:18.751796 23025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:18:19.398195 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.123.148:54973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33515
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:18:19.517611 23060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:18:19.517897 23060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:18:19.518040 23060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:18:19.522164 23060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:18:19.522240 23060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:18:19.522324 23060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:18:19.527787 23060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.123.148:54973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:18:19.528802 23060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:18:19.529850 23060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:18:19.541410 23066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:18:19.541517 23068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:18:19.541832 23065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:18:19.542641 23060 server_base.cc:1034] running on GCE node
I20260620 07:18:19.543123 23060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:18:19.543833 23060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:18:19.545126 23060 hybrid_clock.cc:648] HybridClock initialized: now 1781939899545054 us; error 39 us; skew 500 ppm
Jun 20 07:18:19 dist-test-slave-j663 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:18:19.547919 23060 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:18:19.548872 23060 webserver.cc:460] Webserver started at http://127.11.123.129:34241/ using document root <none> and password file <none>
I20260620 07:18:19.549480 23060 fs_manager.cc:359] Metadata directory not provided
I20260620 07:18:19.549546 23060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:18:19.549758 23060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:18:19 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:18:19.566211 23060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "96c1ed1c43074d4b83be63a1cd71339b"
format_stamp: "Formatted at 2026-06-20 07:18:19 on dist-test-slave-j663"
server_key: "832c9223cad8813d81b18762e34f79d9"
server_key_iv: "c55fd7ec3eba68fd0bea89df79ccf6c9"
server_key_version: "kuduclusterkey@0"
I20260620 07:18:19.567044 23060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "96c1ed1c43074d4b83be63a1cd71339b"
format_stamp: "Formatted at 2026-06-20 07:18:19 on dist-test-slave-j663"
server_key: "832c9223cad8813d81b18762e34f79d9"
server_key_iv: "c55fd7ec3eba68fd0bea89df79ccf6c9"
server_key_version: "kuduclusterkey@0"
I20260620 07:18:19.573362 23060 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 07:18:19.594277 23077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:18:19.595844 23060 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 07:18:19.596033 23060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "96c1ed1c43074d4b83be63a1cd71339b"
format_stamp: "Formatted at 2026-06-20 07:18:19 on dist-test-slave-j663"
server_key: "832c9223cad8813d81b18762e34f79d9"
server_key_iv: "c55fd7ec3eba68fd0bea89df79ccf6c9"
server_key_version: "kuduclusterkey@0"
I20260620 07:18:19.596200 23060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:18:19.614233 23060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:18:19.618000 23060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:18:19.618289 23060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:18:19.619069 23060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:18:19.620131 23060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:18:19.620213 23060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:18:19.620297 23060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:18:19.620335 23060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:18:19.632450 23060 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:35909
I20260620 07:18:19.633818 23060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:18:19.634143 23190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:35909 every 8 connection(s)
I20260620 07:18:19.635916 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 23060
I20260620 07:18:19.636062 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:18:19.636406 11758 external_mini_cluster.cc:1428] Setting key a906b809e0f2ab17ab9bad48c96553f3
Jun 20 07:18:19 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:18:19.652206 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:19.637550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:59171 (local address 127.11.123.190:33515)
0620 07:18:19.638836 (+ 1286us) server_negotiation.cc:206] Beginning negotiation
0620 07:18:19.638872 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0620 07:18:19.638884 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:18:19.638933 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:18:19.638935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:18:19.638975 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:18:19.639103 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:18:19.640285 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:19.640900 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:19.642901 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:19.643375 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:19.643492 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:19.647248 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:18:19.647255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:18:19.647261 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:18:19.647300 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:18:19.649549 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:18:19.650305 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:18:19.650309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:18:19.650311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:18:19.650425 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:18:19.650892 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:18:19.650895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:18:19.650898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:18:19.651125 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:18:19.651257 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 07:18:19.651840 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 07:18:19.651994 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1197,"thread_start_us":119,"threads_started":1}
I20260620 07:18:19.653251 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:19.636683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33515 (local address 127.11.123.129:59171)
0620 07:18:19.637304 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 07:18:19.637364 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 07:18:19.638432 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:18:19.639308 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:18:19.639315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:18:19.639603 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:18:19.640108 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:19.640122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:19.641058 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:19.641066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:19.642744 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:19.642760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:19.643738 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:19.643745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:19.644014 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:19.644836 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:18:19.644864 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:18:19.647014 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:18:19.649727 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:18:19.649736 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:18:19.649759 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:18:19.650162 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:18:19.650546 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:18:19.650551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:18:19.650554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:18:19.650752 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:18:19.651254 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:18:19.651262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:18:19.651564 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 07:18:19.651781 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 07:18:19.652134 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":190,"threads_started":1}
I20260620 07:18:19.654654 23191 heartbeater.cc:347] Connected to a master server at 127.11.123.190:33515
I20260620 07:18:19.654918 23191 heartbeater.cc:464] Registering TS with master...
I20260620 07:18:19.655483 23191 heartbeater.cc:511] Master 127.11.123.190:33515 requested a full tablet report, sending...
I20260620 07:18:19.657238 22979 ts_manager.cc:194] Registered new tserver with Master: 96c1ed1c43074d4b83be63a1cd71339b (127.11.123.129:35909)
I20260620 07:18:19.658551 22979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:59171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:18:19.692672 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.123.148:54973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33515
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:18:19 dist-test-slave-j663 krb5kdc[22503](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 07:18:19.856468 23198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:18:19.856812 23198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:18:19.856881 23198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:18:19.862949 23198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:18:19.863036 23198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:18:19.863124 23198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:18:19.870292 23198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.123.148:54973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:18:19.871430 23198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:18:19.872517 23198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:18:19.884379 23208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:18:19.884514 23205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:18:19.884382 23206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:18:19.885800 23198 server_base.cc:1034] running on GCE node
I20260620 07:18:19.886293 23198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:18:19.887084 23198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:18:19.888298 23198 hybrid_clock.cc:648] HybridClock initialized: now 1781939899888275 us; error 37 us; skew 500 ppm
Jun 20 07:18:19 dist-test-slave-j663 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:18:19.892987 23198 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:18:19.894043 23198 webserver.cc:460] Webserver started at http://127.11.123.130:46819/ using document root <none> and password file <none>
I20260620 07:18:19.894768 23198 fs_manager.cc:359] Metadata directory not provided
I20260620 07:18:19.894892 23198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:18:19.895136 23198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:18:19 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:18:19.912459 23198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "1d4cbd4eb63346ce91fc3fa056811b5a"
format_stamp: "Formatted at 2026-06-20 07:18:19 on dist-test-slave-j663"
server_key: "355bf56e18b33bb677bd88eb115cf847"
server_key_iv: "77f0e968d267cd4d41e713fdba7cb807"
server_key_version: "kuduclusterkey@0"
I20260620 07:18:19.913331 23198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "1d4cbd4eb63346ce91fc3fa056811b5a"
format_stamp: "Formatted at 2026-06-20 07:18:19 on dist-test-slave-j663"
server_key: "355bf56e18b33bb677bd88eb115cf847"
server_key_iv: "77f0e968d267cd4d41e713fdba7cb807"
server_key_version: "kuduclusterkey@0"
I20260620 07:18:19.922605 23198 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260620 07:18:19.939211 23215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:18:19.940893 23198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:18:19.941018 23198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "1d4cbd4eb63346ce91fc3fa056811b5a"
format_stamp: "Formatted at 2026-06-20 07:18:19 on dist-test-slave-j663"
server_key: "355bf56e18b33bb677bd88eb115cf847"
server_key_iv: "77f0e968d267cd4d41e713fdba7cb807"
server_key_version: "kuduclusterkey@0"
I20260620 07:18:19.941257 23198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:18:19.963919 23198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:18:19.967669 23198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:18:19.967912 23198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:18:19.968636 23198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:18:19.969728 23198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:18:19.969832 23198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:18:19.969931 23198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:18:19.970000 23198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:18:19.983312 23198 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:35351
I20260620 07:18:19.984520 23198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:18:19.986208 23328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:35351 every 8 connection(s)
I20260620 07:18:19.993656 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 23198
I20260620 07:18:19.994393 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:18:19.995090 11758 external_mini_cluster.cc:1428] Setting key 1f71df443299119c5d97a2c13b76d26d
Jun 20 07:18:19 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:18:20.004897 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:19.989275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:49535 (local address 127.11.123.190:33515)
0620 07:18:19.989415 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 07:18:19.989419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:18:19.990878 (+ 1459us) server_negotiation.cc:407] Connection header received
0620 07:18:19.992088 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:18:19.992092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:18:19.992146 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:18:19.992198 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:18:19.993858 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:19.995006 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:19.996787 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:19.997194 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:19.997273 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.000564 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:18:20.000568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:18:20.000571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:18:20.000598 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:18:20.002454 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:18:20.003095 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:18:20.003098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:18:20.003100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:18:20.003195 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:18:20.003611 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:18:20.003615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:18:20.003617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:18:20.003834 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:18:20.003903 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 07:18:20.004595 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 07:18:20.004726 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:18:20.006788 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:19.988674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33515 (local address 127.11.123.130:49535)
0620 07:18:19.990718 (+ 2044us) negotiation.cc:107] Waiting for socket to connect
0620 07:18:19.990771 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:18:19.991842 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:18:19.992600 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:18:19.992608 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:18:19.992944 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:18:19.993629 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:19.993649 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:19.995284 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:19.995287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:19.996656 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:19.996668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:19.997468 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:19.997471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:19.997674 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:19.998370 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:18:19.998392 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:18:20.000330 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:18:20.002606 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:18:20.002612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:18:20.002629 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:18:20.002972 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:18:20.003309 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:18:20.003312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:18:20.003314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:18:20.003488 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:18:20.003981 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:18:20.003988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:18:20.004304 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 07:18:20.005396 (+ 1092us) client_negotiation.cc:240] Negotiation successful
0620 07:18:20.005750 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1832,"thread_start_us":137,"threads_started":1}
I20260620 07:18:20.008543 23329 heartbeater.cc:347] Connected to a master server at 127.11.123.190:33515
I20260620 07:18:20.008788 23329 heartbeater.cc:464] Registering TS with master...
I20260620 07:18:20.009377 23329 heartbeater.cc:511] Master 127.11.123.190:33515 requested a full tablet report, sending...
I20260620 07:18:20.010530 22979 ts_manager.cc:194] Registered new tserver with Master: 1d4cbd4eb63346ce91fc3fa056811b5a (127.11.123.130:35351)
I20260620 07:18:20.011302 22979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:49535
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:18:20.060843 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.123.148:54973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33515
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:18:20.222087 23334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:18:20.222486 23334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:18:20.222599 23334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:18:20.228529 23334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:18:20.228648 23334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:18:20.228771 23334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:18:20.236291 23334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.123.148:54973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:33515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:18:20.237702 23334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:18:20.239032 23334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:18:20.250972 23340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:18:20.251008 23342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:18:20.251510 23339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:18:20.252125 23334 server_base.cc:1034] running on GCE node
I20260620 07:18:20.252652 23334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:18:20.253369 23334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:18:20.254550 23334 hybrid_clock.cc:648] HybridClock initialized: now 1781939900254470 us; error 96 us; skew 500 ppm
Jun 20 07:18:20 dist-test-slave-j663 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939900, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:18:20.262921 23334 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:18:20.263948 23334 webserver.cc:460] Webserver started at http://127.11.123.131:39761/ using document root <none> and password file <none>
I20260620 07:18:20.264604 23334 fs_manager.cc:359] Metadata directory not provided
I20260620 07:18:20.264675 23334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:18:20.264891 23334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:18:20 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939900, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
Jun 20 07:18:20 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:18:20.281128 23334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4"
format_stamp: "Formatted at 2026-06-20 07:18:20 on dist-test-slave-j663"
server_key: "f01cc8a5b0981a2d9bcf5a6f985c62b6"
server_key_iv: "a2547a95ce8c43d41aad0414b0fbadca"
server_key_version: "kuduclusterkey@0"
I20260620 07:18:20.281832 23334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4"
format_stamp: "Formatted at 2026-06-20 07:18:20 on dist-test-slave-j663"
server_key: "f01cc8a5b0981a2d9bcf5a6f985c62b6"
server_key_iv: "a2547a95ce8c43d41aad0414b0fbadca"
server_key_version: "kuduclusterkey@0"
I20260620 07:18:20.308561 23334 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.005s sys 0.000s
I20260620 07:18:20.340991 23349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:18:20.342712 23334 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260620 07:18:20.342839 23334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4"
format_stamp: "Formatted at 2026-06-20 07:18:20 on dist-test-slave-j663"
server_key: "f01cc8a5b0981a2d9bcf5a6f985c62b6"
server_key_iv: "a2547a95ce8c43d41aad0414b0fbadca"
server_key_version: "kuduclusterkey@0"
I20260620 07:18:20.342968 23334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:18:20.376389 23334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:18:20.387923 23334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:18:20.388105 23334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:18:20.388774 23334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:18:20.390195 23334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:18:20.390287 23334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:18:20.390478 23334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:18:20.390543 23334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:18:20.400606 23334 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:39349
I20260620 07:18:20.405196 23334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:18:20.411820 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 23334
I20260620 07:18:20.411948 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:18:20.412276 11758 external_mini_cluster.cc:1428] Setting key da36e28f9ab23007b1e57045b276489c
I20260620 07:18:20.407548 23462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:39349 every 8 connection(s)
Jun 20 07:18:20 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939900, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:18:20.425446 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.410182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:34205 (local address 127.11.123.190:33515)
0620 07:18:20.410311 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 07:18:20.410316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:18:20.412118 (+ 1802us) server_negotiation.cc:407] Connection header received
0620 07:18:20.413534 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:18:20.413538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:18:20.413592 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:18:20.413642 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:18:20.415227 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.416052 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.418016 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.418356 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.418424 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.421384 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:18:20.421388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:18:20.421391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:18:20.421416 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:18:20.423136 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:18:20.423712 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:18:20.423715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:18:20.423717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:18:20.423800 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:18:20.424195 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:18:20.424198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:18:20.424200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:18:20.424404 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:18:20.424488 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 07:18:20.425155 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 07:18:20.425288 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:18:20.427078 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.410686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33515 (local address 127.11.123.131:34205)
0620 07:18:20.411902 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0620 07:18:20.412013 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0620 07:18:20.413293 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:18:20.414062 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:18:20.414071 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:18:20.414418 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:18:20.415070 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:20.415088 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.416199 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.416202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:20.417874 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:20.417884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.418609 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.418612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:20.418797 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.419514 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:18:20.419533 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:18:20.421207 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:18:20.423271 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:18:20.423276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:18:20.423290 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:18:20.423602 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:18:20.423907 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:18:20.423910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:18:20.423922 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:18:20.424087 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:18:20.424552 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:18:20.424559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:18:20.424852 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 07:18:20.425790 (+ 938us) client_negotiation.cc:240] Negotiation successful
0620 07:18:20.426112 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":143,"threads_started":1}
I20260620 07:18:20.428383 23463 heartbeater.cc:347] Connected to a master server at 127.11.123.190:33515
I20260620 07:18:20.428603 23463 heartbeater.cc:464] Registering TS with master...
I20260620 07:18:20.429150 23463 heartbeater.cc:511] Master 127.11.123.190:33515 requested a full tablet report, sending...
I20260620 07:18:20.430212 22979 ts_manager.cc:194] Registered new tserver with Master: eb34dbccc5f14dd1b03af5a9a18d24f4 (127.11.123.131:39349)
I20260620 07:18:20.430903 22979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:34205
I20260620 07:18:20.437366 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:18:20.450057 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.439898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.11.123.190:33515)
0620 07:18:20.440015 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:18:20.440019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:18:20.440514 (+ 495us) server_negotiation.cc:407] Connection header received
0620 07:18:20.440677 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:18:20.440680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:18:20.440721 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:18:20.440766 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:18:20.441761 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.442547 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.443736 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.444075 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.444142 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.446220 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:18:20.446224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:18:20.446226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:18:20.446252 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:18:20.448004 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:18:20.448651 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:18:20.448655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:18:20.448657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:18:20.448737 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:18:20.449030 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:18:20.449033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:18:20.449035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:18:20.449284 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:18:20.449385 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:18:20.449772 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 07:18:20.449886 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:18:20.453670 22979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:18:20.457909 22979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:18:20.475579 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.468783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:39349 (local address 127.0.0.1:56076)
0620 07:18:20.469784 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 07:18:20.469826 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:18:20.469956 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:18:20.470832 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:18:20.470838 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:18:20.470859 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:18:20.471007 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:20.471013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.472372 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.472375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:20.473765 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:20.473775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.475079 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.475082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:20.475194 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.475228 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:18:20.475328 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 07:18:20.475437 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":83,"threads_started":1}
I20260620 07:18:20.476781 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.469297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.11.123.131:39349)
0620 07:18:20.470240 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0620 07:18:20.470251 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:18:20.470272 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:18:20.470342 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:18:20.470347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:18:20.470542 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:18:20.470679 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:18:20.471135 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.472243 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.473933 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.474739 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.474852 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.475016 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:18:20.476554 (+ 1538us) server_negotiation.cc:299] Negotiation successful
0620 07:18:20.476651 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":81,"threads_started":1}
I20260620 07:18:20.479957 23397 tablet_service.cc:1467] Processing CreateTablet for tablet 7261ed1566d94df19042b4283f19ae88 (DEFAULT_TABLE table=test-table [id=f6a61a1443854388bcc6dfe6ddf4d4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:18:20.481056 23397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7261ed1566d94df19042b4283f19ae88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:18:20.491272 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.468791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:35351 (local address 127.0.0.1:36092)
0620 07:18:20.475956 (+ 7165us) negotiation.cc:107] Waiting for socket to connect
0620 07:18:20.475970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:18:20.476052 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:18:20.479472 (+ 3420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:18:20.479475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:18:20.479490 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:18:20.480232 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:20.480238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.481844 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.481847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:20.483049 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:20.483059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.490840 (+ 7781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.490843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:20.490961 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.490984 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:18:20.491056 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 07:18:20.491127 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7093,"thread_start_us":112,"threads_started":1}
I20260620 07:18:20.491999 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.469463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.11.123.130:35351)
0620 07:18:20.478890 (+ 9427us) server_negotiation.cc:206] Beginning negotiation
0620 07:18:20.478906 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:18:20.478930 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:18:20.479003 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:18:20.479008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:18:20.479195 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:18:20.479339 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:18:20.480361 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.481716 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.483201 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.483998 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.490647 (+ 6649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.490787 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 07:18:20.491740 (+ 953us) server_negotiation.cc:299] Negotiation successful
0620 07:18:20.491847 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9285,"thread_start_us":76,"threads_started":1}
I20260620 07:18:20.494601 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.469812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:35909 (local address 127.0.0.1:41226)
0620 07:18:20.476372 (+ 6560us) negotiation.cc:107] Waiting for socket to connect
0620 07:18:20.476384 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:18:20.476442 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:18:20.490438 (+ 13996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:18:20.490441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:18:20.490457 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:18:20.490532 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:20.490538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.491898 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.491901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:20.493019 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:20.493029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.494253 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.494256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:20.494334 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.494354 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:18:20.494426 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 07:18:20.494488 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6512,"thread_start_us":89,"threads_started":1}
I20260620 07:18:20.495213 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.470359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.11.123.129:35909)
0620 07:18:20.489851 (+ 19492us) server_negotiation.cc:206] Beginning negotiation
0620 07:18:20.489869 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:18:20.489892 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:18:20.489973 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:18:20.489978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:18:20.490158 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:18:20.490294 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:18:20.490648 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.491765 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.493230 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.493983 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.494092 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.494198 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:18:20.495000 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 07:18:20.495093 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19376,"thread_start_us":98,"threads_started":1}
I20260620 07:18:20.495015 23263 tablet_service.cc:1467] Processing CreateTablet for tablet 7261ed1566d94df19042b4283f19ae88 (DEFAULT_TABLE table=test-table [id=f6a61a1443854388bcc6dfe6ddf4d4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:18:20.495761 23481 tablet_bootstrap.cc:492] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4: Bootstrap starting.
I20260620 07:18:20.496163 23263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7261ed1566d94df19042b4283f19ae88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:18:20.498472 23125 tablet_service.cc:1467] Processing CreateTablet for tablet 7261ed1566d94df19042b4283f19ae88 (DEFAULT_TABLE table=test-table [id=f6a61a1443854388bcc6dfe6ddf4d4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:18:20.499430 23125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7261ed1566d94df19042b4283f19ae88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:18:20.507907 23481 tablet_bootstrap.cc:654] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4: Neither blocks nor log segments found. Creating new log.
I20260620 07:18:20.508716 23481 log.cc:824] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:18:20.509413 23482 tablet_bootstrap.cc:492] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a: Bootstrap starting.
I20260620 07:18:20.511525 23482 tablet_bootstrap.cc:654] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a: Neither blocks nor log segments found. Creating new log.
I20260620 07:18:20.514004 23481 tablet_bootstrap.cc:492] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4: No bootstrap required, opened a new log
I20260620 07:18:20.514242 23481 ts_tablet_manager.cc:1397] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.001s
I20260620 07:18:20.519308 23481 raft_consensus.cc:348] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c1ed1c43074d4b83be63a1cd71339b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 35909 } } peers { permanent_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35351 } } peers { permanent_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 39349 } }
I20260620 07:18:20.520448 23482 log.cc:824] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:18:20.520915 23481 raft_consensus.cc:374] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:18:20.521502 23483 tablet_bootstrap.cc:492] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b: Bootstrap starting.
I20260620 07:18:20.521036 23481 raft_consensus.cc:729] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb34dbccc5f14dd1b03af5a9a18d24f4, State: Initialized, Role: FOLLOWER
I20260620 07:18:20.522223 23481 consensus_queue.cc:260] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c1ed1c43074d4b83be63a1cd71339b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 35909 } } peers { permanent_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35351 } } peers { permanent_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 39349 } }
I20260620 07:18:20.524637 23483 tablet_bootstrap.cc:654] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b: Neither blocks nor log segments found. Creating new log.
I20260620 07:18:20.525664 23481 ts_tablet_manager.cc:1428] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4: Time spent starting tablet: real 0.011s user 0.004s sys 0.001s
I20260620 07:18:20.526087 23463 heartbeater.cc:503] Master 127.11.123.190:33515 was elected leader, sending a full tablet report...
I20260620 07:18:20.531522 23483 log.cc:824] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:18:20.533792 23482 tablet_bootstrap.cc:492] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a: No bootstrap required, opened a new log
I20260620 07:18:20.534075 23482 ts_tablet_manager.cc:1397] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.001s
I20260620 07:18:20.538444 23482 raft_consensus.cc:348] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c1ed1c43074d4b83be63a1cd71339b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 35909 } } peers { permanent_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35351 } } peers { permanent_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 39349 } }
I20260620 07:18:20.540652 23482 raft_consensus.cc:374] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:18:20.540758 23482 raft_consensus.cc:729] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4cbd4eb63346ce91fc3fa056811b5a, State: Initialized, Role: FOLLOWER
I20260620 07:18:20.541340 23482 consensus_queue.cc:260] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c1ed1c43074d4b83be63a1cd71339b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 35909 } } peers { permanent_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35351 } } peers { permanent_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 39349 } }
I20260620 07:18:20.542438 23482 ts_tablet_manager.cc:1428] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20260620 07:18:20.542821 23329 heartbeater.cc:503] Master 127.11.123.190:33515 was elected leader, sending a full tablet report...
I20260620 07:18:20.547036 23483 tablet_bootstrap.cc:492] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b: No bootstrap required, opened a new log
I20260620 07:18:20.547474 23483 ts_tablet_manager.cc:1397] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20260620 07:18:20.552270 23483 raft_consensus.cc:348] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c1ed1c43074d4b83be63a1cd71339b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 35909 } } peers { permanent_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35351 } } peers { permanent_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 39349 } }
I20260620 07:18:20.552618 23483 raft_consensus.cc:374] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:18:20.552717 23483 raft_consensus.cc:729] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96c1ed1c43074d4b83be63a1cd71339b, State: Initialized, Role: FOLLOWER
I20260620 07:18:20.553270 23483 consensus_queue.cc:260] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c1ed1c43074d4b83be63a1cd71339b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 35909 } } peers { permanent_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35351 } } peers { permanent_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 39349 } }
I20260620 07:18:20.554323 23483 ts_tablet_manager.cc:1428] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 07:18:20.554706 23191 heartbeater.cc:503] Master 127.11.123.190:33515 was elected leader, sending a full tablet report...
Jun 20 07:18:20 dist-test-slave-j663 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
W20260620 07:18:20.640301 23192 tablet.cc:2369] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:18:20.645793 23488 raft_consensus.cc:482] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:18:20.646193 23488 raft_consensus.cc:504] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c1ed1c43074d4b83be63a1cd71339b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 35909 } } peers { permanent_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35351 } } peers { permanent_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 39349 } }
I20260620 07:18:20.647722 23488 leader_election.cc:288] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96c1ed1c43074d4b83be63a1cd71339b (127.11.123.129:35909), eb34dbccc5f14dd1b03af5a9a18d24f4 (127.11.123.131:39349)
I20260620 07:18:20.655362 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.649152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:55427 (local address 127.11.123.129:35909)
0620 07:18:20.649279 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:18:20.649283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:18:20.650287 (+ 1004us) server_negotiation.cc:407] Connection header received
0620 07:18:20.650489 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:18:20.650492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:18:20.650540 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:18:20.650590 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:18:20.651017 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.651770 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.653444 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.654025 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.654134 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.654192 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 07:18:20.655142 (+ 950us) server_negotiation.cc:299] Negotiation successful
0620 07:18:20.655220 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:18:20.656414 23492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.648720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:35909 (local address 127.11.123.130:55427)
0620 07:18:20.650209 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0620 07:18:20.650222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:18:20.650351 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:18:20.650729 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:18:20.650732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:18:20.650749 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:18:20.650880 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:20.650887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.651912 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.651916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:20.653280 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:20.653294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.654289 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.654293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:20.654392 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.654431 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 07:18:20.656088 (+ 1657us) client_negotiation.cc:240] Negotiation successful
0620 07:18:20.656232 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1420,"mutex_wait_us":33,"thread_start_us":97,"threads_started":1}
I20260620 07:18:20.657244 23145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7261ed1566d94df19042b4283f19ae88" candidate_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96c1ed1c43074d4b83be63a1cd71339b" is_pre_election: true
I20260620 07:18:20.657593 23145 raft_consensus.cc:2454] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4cbd4eb63346ce91fc3fa056811b5a in term 0.
I20260620 07:18:20.658190 23217 leader_election.cc:302] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4cbd4eb63346ce91fc3fa056811b5a, 96c1ed1c43074d4b83be63a1cd71339b; no voters:
I20260620 07:18:20.658538 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.648639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:39439 (local address 127.11.123.131:39349)
0620 07:18:20.648778 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 07:18:20.648783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:18:20.649953 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 07:18:20.650103 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:18:20.650107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:18:20.650163 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:18:20.650252 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:18:20.652888 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.653759 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.657231 (+ 3472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.657836 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.657937 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.657985 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 07:18:20.658331 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 07:18:20.658430 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:18:20.658495 23488 raft_consensus.cc:2781] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:18:20.658586 23488 raft_consensus.cc:482] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:18:20.658629 23488 raft_consensus.cc:3037] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:18:20.658893 23491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.648139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:39349 (local address 127.11.123.130:39439)
0620 07:18:20.649861 (+ 1722us) negotiation.cc:107] Waiting for socket to connect
0620 07:18:20.649877 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:18:20.649998 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:18:20.652595 (+ 2597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:18:20.652599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:18:20.652619 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:18:20.652738 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:20.652744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.654924 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.654928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:20.656279 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:18:20.656291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.658095 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.658098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:18:20.658198 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.658217 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 07:18:20.658669 (+ 452us) client_negotiation.cc:240] Negotiation successful
0620 07:18:20.658745 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1651,"mutex_wait_us":16,"thread_start_us":153,"threads_started":1}
I20260620 07:18:20.659937 23488 raft_consensus.cc:504] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c1ed1c43074d4b83be63a1cd71339b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 35909 } } peers { permanent_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35351 } } peers { permanent_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 39349 } }
I20260620 07:18:20.660300 23488 leader_election.cc:288] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [CANDIDATE]: Term 1 election: Requested vote from peers 96c1ed1c43074d4b83be63a1cd71339b (127.11.123.129:35909), eb34dbccc5f14dd1b03af5a9a18d24f4 (127.11.123.131:39349)
I20260620 07:18:20.660935 23416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7261ed1566d94df19042b4283f19ae88" candidate_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4"
I20260620 07:18:20.660921 23417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7261ed1566d94df19042b4283f19ae88" candidate_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4" is_pre_election: true
I20260620 07:18:20.661161 23145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7261ed1566d94df19042b4283f19ae88" candidate_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96c1ed1c43074d4b83be63a1cd71339b"
I20260620 07:18:20.661232 23417 raft_consensus.cc:2454] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4cbd4eb63346ce91fc3fa056811b5a in term 0.
I20260620 07:18:20.661283 23145 raft_consensus.cc:3037] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:18:20.662381 23145 raft_consensus.cc:2454] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4cbd4eb63346ce91fc3fa056811b5a in term 1.
I20260620 07:18:20.662725 23217 leader_election.cc:302] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4cbd4eb63346ce91fc3fa056811b5a, 96c1ed1c43074d4b83be63a1cd71339b; no voters:
I20260620 07:18:20.663270 23488 raft_consensus.cc:2781] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:18:20.663488 23488 raft_consensus.cc:686] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 1 LEADER]: Becoming Leader. State: Replica: 1d4cbd4eb63346ce91fc3fa056811b5a, State: Running, Role: LEADER
I20260620 07:18:20.663823 23488 consensus_queue.cc:237] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c1ed1c43074d4b83be63a1cd71339b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 35909 } } peers { permanent_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35351 } } peers { permanent_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 39349 } }
I20260620 07:18:20.667285 22978 catalog_manager.cc:5495] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d4cbd4eb63346ce91fc3fa056811b5a (127.11.123.130). New cstate: current_term: 1 leader_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c1ed1c43074d4b83be63a1cd71339b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 35909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d4cbd4eb63346ce91fc3fa056811b5a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 39349 } health_report { overall_health: UNKNOWN } } }
W20260620 07:18:20.669821 23464 tablet.cc:2369] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:18:20.711944 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.707048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.11.123.130:35351)
0620 07:18:20.707167 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 07:18:20.707172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:18:20.707187 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:18:20.707269 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:18:20.707271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:18:20.707327 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:18:20.707457 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:18:20.707744 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.708569 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.709856 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:18:20.710205 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:18:20.710301 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:18:20.710709 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:18:20.711243 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:18:20.711353 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:18:20.711751 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 07:18:20.711808 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 07:18:20.720223 23416 raft_consensus.cc:3037] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:18:20.721827 23416 raft_consensus.cc:1261] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4 [term 1 FOLLOWER]: Refusing update from remote peer 1d4cbd4eb63346ce91fc3fa056811b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:18:20.722625 23488 consensus_queue.cc:1035] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb34dbccc5f14dd1b03af5a9a18d24f4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:18:20.724742 23145 raft_consensus.cc:1261] T 7261ed1566d94df19042b4283f19ae88 P 96c1ed1c43074d4b83be63a1cd71339b [term 1 FOLLOWER]: Refusing update from remote peer 1d4cbd4eb63346ce91fc3fa056811b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:18:20.725325 23488 consensus_queue.cc:1035] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96c1ed1c43074d4b83be63a1cd71339b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:18:20.740999 23330 tablet.cc:2369] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:18:20.760617 23498 mvcc.cc:205] Tried to move back new op lower bound from 7298825833342357504 to 7298825833121927168. Current Snapshot: MvccSnapshot[applied={T|T < 7298825833342357504}]
I20260620 07:18:20.761049 23500 mvcc.cc:205] Tried to move back new op lower bound from 7298825833342357504 to 7298825833121927168. Current Snapshot: MvccSnapshot[applied={T|T < 7298825833342357504}]
I20260620 07:18:20.771329 22978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:18:20.771555 22978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:18:20.774677 22978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0109103522d14b599215204bb8cf4876: Sending DeleteTablet for 3 replicas of tablet 7261ed1566d94df19042b4283f19ae88
I20260620 07:18:20.775660 23263 tablet_service.cc:1514] Processing DeleteTablet for tablet 7261ed1566d94df19042b4283f19ae88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:18:20 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:36092
I20260620 07:18:20.775998 23397 tablet_service.cc:1514] Processing DeleteTablet for tablet 7261ed1566d94df19042b4283f19ae88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:18:20 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:56076
I20260620 07:18:20.776409 23125 tablet_service.cc:1514] Processing DeleteTablet for tablet 7261ed1566d94df19042b4283f19ae88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:18:20 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:41226
I20260620 07:18:20.778146 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 23060
I20260620 07:18:20.783699 23507 tablet_replica.cc:330] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4: stopping tablet replica
I20260620 07:18:20.793450 23507 raft_consensus.cc:2229] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:18:20.793989 23507 raft_consensus.cc:2258] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:18:20.794713 23506 tablet_replica.cc:330] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a: stopping tablet replica
I20260620 07:18:20.795081 23506 raft_consensus.cc:2229] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:18:20.795532 23506 raft_consensus.cc:2258] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:18:20.795828 23507 ts_tablet_manager.cc:1905] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:18:20.796943 23506 ts_tablet_manager.cc:1905] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:18:20.798547 23507 ts_tablet_manager.cc:1918] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:18:20.798629 23507 log.cc:1196] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/7261ed1566d94df19042b4283f19ae88
I20260620 07:18:20.799335 23507 ts_tablet_manager.cc:1939] T 7261ed1566d94df19042b4283f19ae88 P eb34dbccc5f14dd1b03af5a9a18d24f4: Deleting consensus metadata
I20260620 07:18:20.799494 23506 ts_tablet_manager.cc:1918] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:18:20.799571 23506 log.cc:1196] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/7261ed1566d94df19042b4283f19ae88
I20260620 07:18:20.799849 23506 ts_tablet_manager.cc:1939] T 7261ed1566d94df19042b4283f19ae88 P 1d4cbd4eb63346ce91fc3fa056811b5a: Deleting consensus metadata
I20260620 07:18:20.800624 22963 catalog_manager.cc:4841] TS eb34dbccc5f14dd1b03af5a9a18d24f4 (127.11.123.131:39349): tablet 7261ed1566d94df19042b4283f19ae88 (table test-table [id=f6a61a1443854388bcc6dfe6ddf4d4ce]) successfully deleted
I20260620 07:18:20.800837 22966 catalog_manager.cc:4841] TS 1d4cbd4eb63346ce91fc3fa056811b5a (127.11.123.130:35351): tablet 7261ed1566d94df19042b4283f19ae88 (table test-table [id=f6a61a1443854388bcc6dfe6ddf4d4ce]) successfully deleted
W20260620 07:18:20.801281 22964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:35909 (error 108)
I20260620 07:18:20.802661 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 23198
I20260620 07:18:20.804114 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:18:20.803810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:35909 (local address 127.0.0.1:41240)
0620 07:18:20.803942 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 07:18:20.804029 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:35909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 07:18:20.804282 22964 catalog_manager.cc:4584] TS 96c1ed1c43074d4b83be63a1cd71339b (127.11.123.129:35909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7261ed1566d94df19042b4283f19ae88: Network error: Client connection negotiation failed: client connection to 127.11.123.129:35909: connect: Connection refused (error 111)
I20260620 07:18:20.813727 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 23334
I20260620 07:18:20.820675 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 22947
I20260620 07:18:20.827651 11758 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 07:18:21.172092 11758 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 07:18:21.172204 11758 mini_ranger.cc:66] Stopping Ranger...
I20260620 07:18:21.508262 11758 mini_ranger.cc:68] Stopped Ranger
2026-06-20 07:18:21.508 UTC [22533] LOG: received smart shutdown request
2026-06-20 07:18:21.511 UTC [22533] LOG: background worker "logical replication launcher" (PID 22542) exited with exit code 1
2026-06-20 07:18:21.512 UTC [22537] LOG: shutting down
2026-06-20 07:18:21.671 UTC [22533] LOG: database system is shut down
2026-06-20T07:18:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44594 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:18:21 dist-test-slave-j663 krb5kdc[23518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:18:21 dist-test-slave-j663 krb5kdc[23518](info): set up 2 sockets
Jun 20 07:18:21 dist-test-slave-j663 krb5kdc[23518](info): commencing operation
krb5kdc: starting...
W20260620 07:18:22.837189 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:18:22 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:18:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.123.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.123.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/rangeradmin_127.11.123.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/rangeradmin_127.11.123.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.123.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.123.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/rangerlookup_127.11.123.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/rangerlookup_127.11.123.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.123.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.123.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/HTTP_127.11.123.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/HTTP_127.11.123.148@KRBTEST.COM.keytab.
I20260620 07:18:23.008054 11758 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbzqqUD/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/postgres -l logfile start
2026-06-20 07:18:25.645 UTC [23548] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 07:18:25.645 UTC [23548] LOG: listening on IPv4 address "127.11.123.148", port 47647
2026-06-20 07:18:25.649 UTC [23548] LOG: listening on Unix socket "/tmp/.s.PGSQL.47647"
2026-06-20 07:18:25.684 UTC [23551] LOG: database system was shut down at 2026-06-20 07:18:24 UTC
2026-06-20 07:18:25.688 UTC [23548] LOG: database system is ready to accept connections
I20260620 07:18:26.677067 11758 mini_postgres.cc:96] Postgres bound to 47647
2026-06-20 07:18:26.683 UTC [23559] FATAL: database "slave" does not exist
127.11.123.148:47647 - accepting connections
I20260620 07:18:26.683970 11758 mini_ranger.cc:161] Starting Ranger...
I20260620 07:18:26.698493 11758 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 07:18:27.246500 11758 mini_ranger.cc:87] Created ranger Postgres database
I20260620 07:18:27.246589 11758 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-admin
2026-06-20 07:18:27,519 [I] DB FLAVOR :POSTGRES
2026-06-20 07:18:27,520 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:18:27,520 [I] Checking connection..
2026-06-20 07:18:27,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 07:18:27,780 [I] Checking connection passed.
2026-06-20 07:18:27,780 [I] --------- Verifying version history table ---------
2026-06-20 07:18:27,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:18:28,018 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:18:28,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:18:28,252 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:18:28,253 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 07:18:28,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 07:18:28.505 UTC [23623] WARNING: there is no transaction in progress
2026-06-20 07:18:28,511 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 07:18:28,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:18:28,757 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 07:18:28,757 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:18:28,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 07:18:28,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 07:18:29,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j663.c.gcp-upstream.internal','N') ;"
2026-06-20 07:18:29,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:18:29,686 [I] Table x_portal_user does not exist in database ranger
2026-06-20 07:18:29,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:18:29,934 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 07:18:29,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:18:30,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:18:30,404 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 07:18:30,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 07:18:31.361 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.377 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.393 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.409 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.425 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.442 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.498 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.505 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.520 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.530 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.540 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.555 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.565 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.572 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.580 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:31.587 UTC [23731] WARNING: there is no transaction in progress
2026-06-20 07:18:32,273 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 07:18:32,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 07:18:32,499 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 07:18:32,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:18:32,731 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 07:18:32,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:18:32,988 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 07:18:32,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:18:33,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:18:33,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j663.c.gcp-upstream.internal';"
2026-06-20 07:18:33,710 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 07:18:33,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 07:18:33,961 [I] DB_PATCHES have already been applied
I20260620 07:18:33.967209 11758 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbzqqUD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 07:18:33.967304 11758 mini_ranger.cc:193] Using host: 127.11.123.148
I20260620 07:18:33.970750 11758 mini_ranger.cc:227] Ranger admin URL: http://127.11.123.148:55569
Jun 20, 2026 7:18:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:18:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:18:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:18:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:18:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:18:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:18:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 07:18:35.076370 11758 mini_ranger.cc:160] Time spent starting Ranger: real 8.392s user 0.002s sys 0.008s
Jun 20, 2026 7:18:55 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:18:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 7:18:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:18:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:18:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:18:56 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 07:18:56.767660 11758 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.123.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.123.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/rangerkms_127.11.123.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.123.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/rangerkms_127.11.123.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.123.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.123.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.123.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/HTTP_127.11.123.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.123.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/HTTP_127.11.123.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 07:18:56 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939936, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 07:18:56.850623 11758 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 07:18:56.859867 11758 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 07:18:57.676654 11758 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 07:18:57.676753 11758 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-kms
I20260620 07:18:57.676775 11758 mini_ranger_kms.cc:227] Using postgres at 127.11.123.148:47647
2026-06-20 07:18:57,851 [I] DB FLAVOR :POSTGRES
2026-06-20 07:18:57,851 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:18:57,851 [I] Checking connection
2026-06-20 07:18:57,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 07:18:58,097 [I] connection success
2026-06-20 07:18:58,098 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 07:18:58,098 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 07:18:58,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 07:18:58,322 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 07:18:58,323 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:18:58,323 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 07:18:58,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.123.148:47647/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 07:18:58,596 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 07:18:59.072335 11758 mini_ranger_kms.cc:310] Created kms service
I20260620 07:18:59.248009 11758 mini_ranger_kms.cc:326] Created kudu user
I20260620 07:18:59.288843 11758 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 07:18:59 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939936, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:18:59.637537 11758 mini_ranger_kms.cc:384] Added ranger policy
I20260620 07:18:59.637687 11758 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbzqqUD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbzqqUD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 07:18:59.637709 11758 mini_ranger_kms.cc:240] Using host: 127.11.123.148
I20260620 07:18:59.642094 11758 mini_ranger_kms.cc:276] Ranger KMS PID: 23911
I20260620 07:18:59.642194 11758 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.123.148:49705
Jun 20, 2026 7:19:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:19:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:19:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:19:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:19:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:19:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 07:19:01.900857 11758 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.050s user 0.010s sys 0.148s
I20260620 07:19:01.900991 11758 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 07:19:01.901057 11758 mini_ranger_kms.cc:401] 127.11.123.148:49705/kms/v1/keys
Jun 20 07:19:02 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939942, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.123.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 07:19:02 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939942, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.123.148@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
Jun 20 07:19:02 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939942, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.123.148@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
Jun 20 07:19:04 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:19:04.976256 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:45703
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:45703
--ranger_config_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.123.148:49705/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:05.097179 23962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:05.097450 23962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:05.097533 23962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:05.101491 23962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:19:05.101568 23962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:05.101605 23962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:05.101642 23962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:19:05.101671 23962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:19:05.106936 23962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.123.148:49705/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:45703
--ranger_config_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:45703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:05.107909 23962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:05.108878 23962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:05.118052 23967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:05.118057 23968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:05.118104 23970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:05.118433 23962 server_base.cc:1034] running on GCE node
I20260620 07:19:05.119665 23962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:05.121155 23962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:05.122394 23962 hybrid_clock.cc:648] HybridClock initialized: now 1781939945122352 us; error 39 us; skew 500 ppm
Jun 20 07:19:05 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939945, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:05.129338 23962 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:19:05.130296 23962 webserver.cc:460] Webserver started at http://127.11.123.190:37285/ using document root <none> and password file <none>
I20260620 07:19:05.130839 23962 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:05.130916 23962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:05.131127 23962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:19:05 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939945, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:19:05.189253 23962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "5877b27ebe22481b81c102718a43a6dc"
format_stamp: "Formatted at 2026-06-20 07:19:05 on dist-test-slave-j663"
server_key: "51c2e49f84eee77cf475169824069120"
server_key_iv: "1a0e32bdad0e0f2f4bb8d4fae4e16ddf"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:05.189878 23962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "5877b27ebe22481b81c102718a43a6dc"
format_stamp: "Formatted at 2026-06-20 07:19:05 on dist-test-slave-j663"
server_key: "51c2e49f84eee77cf475169824069120"
server_key_iv: "1a0e32bdad0e0f2f4bb8d4fae4e16ddf"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:05.193746 23962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:19:05.207031 23977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:05.208379 23962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:19:05.208514 23962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "5877b27ebe22481b81c102718a43a6dc"
format_stamp: "Formatted at 2026-06-20 07:19:05 on dist-test-slave-j663"
server_key: "51c2e49f84eee77cf475169824069120"
server_key_iv: "1a0e32bdad0e0f2f4bb8d4fae4e16ddf"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:05.208659 23962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:05.230800 23962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:05.233760 23962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:05.234228 23962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:05.241894 23962 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:45703
I20260620 07:19:05.241912 24029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:45703 every 8 connection(s)
I20260620 07:19:05.242884 23962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:19:05.243691 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 23962
I20260620 07:19:05.243822 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:19:05.244099 11758 external_mini_cluster.cc:1428] Setting key 7be8ceb5aec4cd56de5f3cb20e2cbb0a
I20260620 07:19:05.246162 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:19:05 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:05.252789 24030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc: Bootstrap starting.
I20260620 07:19:05.255494 24030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:05.256754 24030 log.cc:824] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:05.257782 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:05.245499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.11.123.190:45703)
0620 07:19:05.245877 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:05.245894 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:05.245939 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:19:05.246681 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:05.246698 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:05.247004 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:05.247341 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:05.248107 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:05.249065 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:05.249950 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:05.250412 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:05.250585 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:05.252986 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:05.252991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:05.253003 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:05.253028 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:05.254817 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:05.255246 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:05.255252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:05.255258 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:05.255365 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:05.255672 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:05.255675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:05.255677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:05.256030 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:05.256224 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:05.256506 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 07:19:05.256910 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":116,"threads_started":1}
I20260620 07:19:05.258867 24030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc: No bootstrap required, opened a new log
I20260620 07:19:05.262508 24030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5877b27ebe22481b81c102718a43a6dc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45703 } }
I20260620 07:19:05.262761 24030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:05.262812 24030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5877b27ebe22481b81c102718a43a6dc, State: Initialized, Role: FOLLOWER
I20260620 07:19:05.263226 24030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5877b27ebe22481b81c102718a43a6dc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45703 } }
I20260620 07:19:05.263367 24030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:19:05.263453 24030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:19:05.263541 24030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:05.264534 24030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5877b27ebe22481b81c102718a43a6dc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45703 } }
I20260620 07:19:05.264873 24030 leader_election.cc:302] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5877b27ebe22481b81c102718a43a6dc; no voters:
I20260620 07:19:05.265177 24030 leader_election.cc:288] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:19:05.265314 24035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:19:05.265610 24035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 1 LEADER]: Becoming Leader. State: Replica: 5877b27ebe22481b81c102718a43a6dc, State: Running, Role: LEADER
I20260620 07:19:05.265921 24035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5877b27ebe22481b81c102718a43a6dc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45703 } }
I20260620 07:19:05.266361 24030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:19:05.266933 24035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5877b27ebe22481b81c102718a43a6dc. Latest consensus state: current_term: 1 leader_uuid: "5877b27ebe22481b81c102718a43a6dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5877b27ebe22481b81c102718a43a6dc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45703 } } }
I20260620 07:19:05.267143 24035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:05.267313 24036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5877b27ebe22481b81c102718a43a6dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5877b27ebe22481b81c102718a43a6dc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45703 } } }
I20260620 07:19:05.267465 24036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:05.267627 24044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:19:05.270956 24044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:19:05.271394 24030 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 07:19:05.280980 24044 catalog_manager.cc:1347] Generated new cluster ID: d53d89d2b2ab40f8a9733a2501485cf6
I20260620 07:19:05.281147 24044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:19:05.298194 24044 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:19:05.299479 24044 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:19:05.309437 24044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc: Generated new TSK 0
I20260620 07:19:05.310292 24044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:19:06.068174 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.123.148:49705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:45703
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:06.181416 24075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:06.181792 24075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:06.181903 24075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:06.186089 24075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:06.186189 24075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:06.186288 24075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:19:06.191804 24075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.123.148:49705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:06.192893 24075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:06.193935 24075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:06.204262 24083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:06.204306 24080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:06.204345 24081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:06.204546 24075 server_base.cc:1034] running on GCE node
I20260620 07:19:06.204998 24075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:06.205668 24075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:06.206856 24075 hybrid_clock.cc:648] HybridClock initialized: now 1781939946206828 us; error 44 us; skew 500 ppm
Jun 20 07:19:06 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939946, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:06.209882 24075 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:19:06.210765 24075 webserver.cc:460] Webserver started at http://127.11.123.129:35967/ using document root <none> and password file <none>
I20260620 07:19:06.211434 24075 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:06.211529 24075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:06.211766 24075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:19:06 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939946, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:19:06.229810 24075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d"
format_stamp: "Formatted at 2026-06-20 07:19:06 on dist-test-slave-j663"
server_key: "db7b8efe017e254f3ad0da3e5ef124de"
server_key_iv: "2f52331f54dbf04194e1fe3d7019b9a0"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:06.230609 24075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d"
format_stamp: "Formatted at 2026-06-20 07:19:06 on dist-test-slave-j663"
server_key: "db7b8efe017e254f3ad0da3e5ef124de"
server_key_iv: "2f52331f54dbf04194e1fe3d7019b9a0"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:06.235621 24075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:19:06.253734 24092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:06.255380 24075 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 07:19:06.255503 24075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d"
format_stamp: "Formatted at 2026-06-20 07:19:06 on dist-test-slave-j663"
server_key: "db7b8efe017e254f3ad0da3e5ef124de"
server_key_iv: "2f52331f54dbf04194e1fe3d7019b9a0"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:06.255626 24075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:06.284368 24075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:06.287894 24075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:06.288146 24075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:06.288817 24075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:06.289920 24075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:06.290040 24075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:06.290174 24075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:06.290266 24075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:06.300557 24075 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:45673
I20260620 07:19:06.301756 24075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:19:06.308040 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 24075
I20260620 07:19:06.308167 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:19:06.308468 11758 external_mini_cluster.cc:1428] Setting key f151a4d42b540f6510faf01474db0ef4
I20260620 07:19:06.314684 24205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:45673 every 8 connection(s)
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
Jun 20 07:19:06 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939946, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:06.340956 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:06.318266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:40797 (local address 127.11.123.190:45703)
0620 07:19:06.322985 (+ 4719us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:06.322990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:06.323007 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:19:06.323061 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:06.323064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:06.323112 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:06.323842 (+ 730us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:06.326287 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:06.327270 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:06.329343 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:06.329750 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:06.329867 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:06.336400 (+ 6533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:06.336404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:06.336407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:06.336435 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:06.338525 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:06.339144 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:06.339148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:06.339150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:06.339244 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:06.339660 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:06.339663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:06.339665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:06.339891 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:06.339974 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:06.340644 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 07:19:06.340772 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4638,"thread_start_us":84,"threads_started":1}
I20260620 07:19:06.342960 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:06.317144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45703 (local address 127.11.123.129:40797)
0620 07:19:06.321319 (+ 4175us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:06.321385 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:06.322642 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:06.323554 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:06.323565 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:06.325304 (+ 1739us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:06.326052 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:06.326073 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:06.327430 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:06.327434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:06.329186 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:06.329199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:06.330065 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:06.330070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:06.330287 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:06.332850 (+ 2563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:06.332871 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:06.336185 (+ 3314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:06.338673 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:06.338680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:06.338697 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:06.339023 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:06.339353 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:06.339356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:06.339358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:06.339527 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:06.340041 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:06.340047 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:06.340348 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 07:19:06.341440 (+ 1092us) client_negotiation.cc:240] Negotiation successful
0620 07:19:06.341836 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3878,"thread_start_us":145,"threads_started":1}
I20260620 07:19:06.344880 24206 heartbeater.cc:347] Connected to a master server at 127.11.123.190:45703
I20260620 07:19:06.345152 24206 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:06.345715 24206 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
I20260620 07:19:06.347453 23994 ts_manager.cc:194] Registered new tserver with Master: ddc9afc87dd34e0e8b3a1f8256e3dc8d (127.11.123.129:45673)
I20260620 07:19:06.349020 23994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:40797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:19:06.350759 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.123.148:49705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:45703
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:19:06 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939946, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 07:19:06.505307 24214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:06.505692 24214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:06.505764 24214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:06.511765 24214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:06.511951 24214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:06.512106 24214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:19:06.520066 24214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.123.148:49705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:06.521387 24214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:06.522703 24214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:06.534785 24223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:06.534999 24220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:06.534817 24221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:06.535595 24214 server_base.cc:1034] running on GCE node
I20260620 07:19:06.535990 24214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:06.536677 24214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:06.537853 24214 hybrid_clock.cc:648] HybridClock initialized: now 1781939946537779 us; error 86 us; skew 500 ppm
Jun 20 07:19:06 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939946, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:06.541356 24214 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:19:06.542337 24214 webserver.cc:460] Webserver started at http://127.11.123.130:33155/ using document root <none> and password file <none>
I20260620 07:19:06.542984 24214 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:06.543071 24214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:06.543283 24214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:19:06 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939946, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:19:06.561172 24214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "84add220926047f68faf83b37de8372d"
format_stamp: "Formatted at 2026-06-20 07:19:06 on dist-test-slave-j663"
server_key: "14220e5a8c9fae697db577d7bbc42c71"
server_key_iv: "3940105dd825e6d37f542bf940315107"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:06.561923 24214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "84add220926047f68faf83b37de8372d"
format_stamp: "Formatted at 2026-06-20 07:19:06 on dist-test-slave-j663"
server_key: "14220e5a8c9fae697db577d7bbc42c71"
server_key_iv: "3940105dd825e6d37f542bf940315107"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:06.566872 24214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 07:19:06.586027 24230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:06.589148 24214 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260620 07:19:06.589345 24214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "84add220926047f68faf83b37de8372d"
format_stamp: "Formatted at 2026-06-20 07:19:06 on dist-test-slave-j663"
server_key: "14220e5a8c9fae697db577d7bbc42c71"
server_key_iv: "3940105dd825e6d37f542bf940315107"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:06.589552 24214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:06.621332 24214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:06.624785 24214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:06.624969 24214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:06.625603 24214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:06.626479 24214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:06.626547 24214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:06.626667 24214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:06.626711 24214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:06.637672 24214 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:41153
I20260620 07:19:06.637706 24343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:41153 every 8 connection(s)
I20260620 07:19:06.638639 24214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:19:06.644375 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 24214
I20260620 07:19:06.644497 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:19:06.644795 11758 external_mini_cluster.cc:1428] Setting key 3e082470a6b58443579f5dfd91ee065b
Jun 20 07:19:06 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939946, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:06.652931 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:06.640511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:38955 (local address 127.11.123.190:45703)
0620 07:19:06.640656 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:06.640661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:06.641458 (+ 797us) server_negotiation.cc:407] Connection header received
0620 07:19:06.642434 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:06.642438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:06.642493 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:06.642588 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:06.643616 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:06.644198 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:06.645799 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:06.646122 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:06.646207 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:06.649355 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:06.649359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:06.649362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:06.649395 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:06.650882 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:06.651403 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:06.651406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:06.651408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:06.651488 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:06.651832 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:06.651836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:06.651837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:06.652013 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:06.652117 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:06.652630 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 07:19:06.652761 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:19:06.653825 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:06.640868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45703 (local address 127.11.123.130:38955)
0620 07:19:06.641310 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:06.641362 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:06.642225 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:06.642762 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:06.642769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:06.643024 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:06.643479 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:06.643491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:06.644356 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:06.644359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:06.645676 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:06.645685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:06.646400 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:06.646404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:06.646583 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:06.647278 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:06.647303 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:06.649146 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:06.651018 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:06.651024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:06.651040 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:06.651295 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:06.651583 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:06.651586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:06.651588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:06.651733 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:06.652115 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:06.652122 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:06.652364 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 07:19:06.652550 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 07:19:06.652877 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":124,"threads_started":1}
I20260620 07:19:06.655108 24344 heartbeater.cc:347] Connected to a master server at 127.11.123.190:45703
I20260620 07:19:06.655372 24344 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:06.655869 24344 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
I20260620 07:19:06.656981 23994 ts_manager.cc:194] Registered new tserver with Master: 84add220926047f68faf83b37de8372d (127.11.123.130:41153)
I20260620 07:19:06.657637 23994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:38955
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:19:06.688989 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.123.148:49705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:45703
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:06.856101 24349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:06.856519 24349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:06.856633 24349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:06.863413 24349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:06.863545 24349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:06.863672 24349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:19:06.871795 24349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.123.148:49705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:06.873304 24349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:06.874672 24349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 07:19:06 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939946, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
W20260620 07:19:06.887059 24357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:06.887507 24354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:06.888098 24355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:06.889142 24349 server_base.cc:1034] running on GCE node
I20260620 07:19:06.889580 24349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:06.890275 24349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:06.900151 24349 hybrid_clock.cc:648] HybridClock initialized: now 1781939946900062 us; error 98 us; skew 500 ppm
Jun 20 07:19:06 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939946, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:06.903858 24349 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:19:06.905041 24349 webserver.cc:460] Webserver started at http://127.11.123.131:41447/ using document root <none> and password file <none>
I20260620 07:19:06.907738 24349 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:06.909300 24349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:06.909649 24349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:19:06 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939946, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:19:06.938326 24349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "f591e9c45cf34128baf171a5bf07f913"
format_stamp: "Formatted at 2026-06-20 07:19:06 on dist-test-slave-j663"
server_key: "24c547e24bff949d0d2af4e4a889757a"
server_key_iv: "8a944ee507bfd3371781d3dfd6753679"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:06.940064 24349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "f591e9c45cf34128baf171a5bf07f913"
format_stamp: "Formatted at 2026-06-20 07:19:06 on dist-test-slave-j663"
server_key: "24c547e24bff949d0d2af4e4a889757a"
server_key_iv: "8a944ee507bfd3371781d3dfd6753679"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:06.945127 24349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 07:19:06.966807 24364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:06.968454 24349 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:19:06.968577 24349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "f591e9c45cf34128baf171a5bf07f913"
format_stamp: "Formatted at 2026-06-20 07:19:06 on dist-test-slave-j663"
server_key: "24c547e24bff949d0d2af4e4a889757a"
server_key_iv: "8a944ee507bfd3371781d3dfd6753679"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:06.968695 24349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:06.989573 24349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:06.993165 24349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:06.993393 24349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:06.994038 24349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:06.995048 24349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:06.995213 24349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:06.995321 24349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:06.995384 24349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:07.007378 24349 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:44783
I20260620 07:19:07.008437 24349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:19:07.010349 24477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:44783 every 8 connection(s)
I20260620 07:19:07.016539 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 24349
I20260620 07:19:07.016666 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:19:07.016925 11758 external_mini_cluster.cc:1428] Setting key 0eef6dc861d5beb72700dece82a35f50
Jun 20 07:19:07 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939946, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:07.024935 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:07.012047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:45539 (local address 127.11.123.190:45703)
0620 07:19:07.012184 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:07.012189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:07.013409 (+ 1220us) server_negotiation.cc:407] Connection header received
0620 07:19:07.014592 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:07.014596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:07.014675 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:07.014775 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:07.016120 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:07.016983 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:07.018581 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:07.018910 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:07.019001 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:07.021467 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:07.021471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:07.021474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:07.021504 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:07.022918 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:07.023429 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:07.023432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:07.023434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:07.023525 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:07.023867 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:07.023871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:07.023872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:07.024054 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:07.024161 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:07.024645 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 07:19:07.024770 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 07:19:07.025934 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:07.012583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45703 (local address 127.11.123.131:45539)
0620 07:19:07.013257 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:07.013313 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:07.014312 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:07.014983 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:07.014992 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:07.015324 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:07.015943 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:07.015972 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:07.017119 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:07.017123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:07.018431 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:07.018440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:07.019190 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:07.019192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:07.019329 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:07.019948 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:07.019970 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:07.021312 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:07.023052 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:07.023058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:07.023073 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:07.023324 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:07.023620 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:07.023624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:07.023626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:07.023769 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:07.024158 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:07.024165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:07.024407 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 07:19:07.024593 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 07:19:07.024891 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":148,"threads_started":1}
I20260620 07:19:07.028626 24478 heartbeater.cc:347] Connected to a master server at 127.11.123.190:45703
I20260620 07:19:07.028859 24478 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:07.029481 24478 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
I20260620 07:19:07.030675 23994 ts_manager.cc:194] Registered new tserver with Master: f591e9c45cf34128baf171a5bf07f913 (127.11.123.131:44783)
I20260620 07:19:07.031245 23994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:45539
I20260620 07:19:07.042534 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:19:07.043295 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 24075
I20260620 07:19:07.052547 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 24214
I20260620 07:19:07.060264 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 24349
I20260620 07:19:07.068244 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 23962
I20260620 07:19:07.078976 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:45703
--webserver_interface=127.11.123.190
--webserver_port=37285
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:45703
--ranger_config_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.123.148:49705/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.123.148:49705/kms with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:07.187749 24481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:07.188002 24481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:07.188040 24481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:07.192088 24481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:19:07.192147 24481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:07.192166 24481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:07.192183 24481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:19:07.192302 24481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:19:07.197834 24481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.123.148:49705/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:45703
--ranger_config_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:45703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=37285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:07.198832 24481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:07.199803 24481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:07.209172 24486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:07.209231 24489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:07.209435 24481 server_base.cc:1034] running on GCE node
W20260620 07:19:07.209488 24487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:07.210171 24481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:07.211481 24481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:07.212684 24481 hybrid_clock.cc:648] HybridClock initialized: now 1781939947212647 us; error 44 us; skew 500 ppm
Jun 20 07:19:07 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939947, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:07.215833 24481 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:19:07.216686 24481 webserver.cc:460] Webserver started at http://127.11.123.190:37285/ using document root <none> and password file <none>
I20260620 07:19:07.217257 24481 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:07.217324 24481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:19:07 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939947, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:19:07.300689 24481 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:19:07.303048 24497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:07.304330 24481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:19:07.304471 24481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "5877b27ebe22481b81c102718a43a6dc"
format_stamp: "Formatted at 2026-06-20 07:19:05 on dist-test-slave-j663"
server_key: "51c2e49f84eee77cf475169824069120"
server_key_iv: "1a0e32bdad0e0f2f4bb8d4fae4e16ddf"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:07.305012 24481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:07.324321 24481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:07.327330 24481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:07.327538 24481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:07.334687 24481 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:45703
I20260620 07:19:07.334697 24549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:45703 every 8 connection(s)
I20260620 07:19:07.335762 24481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:19:07.339879 24550 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:19:07.341480 24550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc: Bootstrap starting.
I20260620 07:19:07.345894 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 24481
I20260620 07:19:07.346338 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.123.148:49705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:45673
--local_ip_for_outbound_sockets=127.11.123.129
--tserver_master_addrs=127.11.123.190:45703
--webserver_port=35967
--webserver_interface=127.11.123.129
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.123.148:49705/kms with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:19:07.357409 24550 log.cc:824] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:07.363972 24550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:19:07.364370 24550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc: Bootstrap complete.
I20260620 07:19:07.368439 24550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5877b27ebe22481b81c102718a43a6dc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45703 } }
I20260620 07:19:07.368847 24550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5877b27ebe22481b81c102718a43a6dc, State: Initialized, Role: FOLLOWER
I20260620 07:19:07.369366 24550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5877b27ebe22481b81c102718a43a6dc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45703 } }
I20260620 07:19:07.369509 24550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:19:07.369599 24550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:19:07.369712 24550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:19:07.371567 24550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5877b27ebe22481b81c102718a43a6dc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45703 } }
I20260620 07:19:07.371891 24550 leader_election.cc:302] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5877b27ebe22481b81c102718a43a6dc; no voters:
I20260620 07:19:07.372210 24550 leader_election.cc:288] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:19:07.372330 24554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:19:07.372560 24554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [term 2 LEADER]: Becoming Leader. State: Replica: 5877b27ebe22481b81c102718a43a6dc, State: Running, Role: LEADER
I20260620 07:19:07.372856 24554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5877b27ebe22481b81c102718a43a6dc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45703 } }
I20260620 07:19:07.373435 24550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:19:07.373776 24556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5877b27ebe22481b81c102718a43a6dc. Latest consensus state: current_term: 2 leader_uuid: "5877b27ebe22481b81c102718a43a6dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5877b27ebe22481b81c102718a43a6dc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45703 } } }
I20260620 07:19:07.373889 24556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:07.374140 24555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5877b27ebe22481b81c102718a43a6dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5877b27ebe22481b81c102718a43a6dc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45703 } } }
I20260620 07:19:07.374223 24558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:19:07.374238 24555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:07.377380 24558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:19:07.377950 24550 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 07:19:07.377985 24558 catalog_manager.cc:1259] Loaded cluster ID: d53d89d2b2ab40f8a9733a2501485cf6
I20260620 07:19:07.378129 24558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:19:07.380473 24558 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:19:07.381119 24558 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc: Loaded TSK: 0
I20260620 07:19:07.382302 24558 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:19:07.489171 24552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:07.489487 24552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:07.489548 24552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:07.495728 24552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:07.495810 24552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:07.495898 24552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:19:07.504025 24552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.123.148:49705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:45673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=35967
--tserver_master_addrs=127.11.123.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:07.505477 24552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:07.506850 24552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:07.519421 24590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:07.519750 24587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:07.520026 24588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:07.520174 24552 server_base.cc:1034] running on GCE node
I20260620 07:19:07.520797 24552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:07.521647 24552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:07.529054 24552 hybrid_clock.cc:648] HybridClock initialized: now 1781939947529024 us; error 47 us; skew 500 ppm
Jun 20 07:19:07 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939947, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:07.532617 24552 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:19:07.533715 24552 webserver.cc:460] Webserver started at http://127.11.123.129:35967/ using document root <none> and password file <none>
I20260620 07:19:07.534276 24552 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:07.534389 24552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:19:07 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939947, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:19:07.558619 24552 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 07:19:07.561705 24598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:07.563146 24552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:19:07.563285 24552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d"
format_stamp: "Formatted at 2026-06-20 07:19:06 on dist-test-slave-j663"
server_key: "db7b8efe017e254f3ad0da3e5ef124de"
server_key_iv: "2f52331f54dbf04194e1fe3d7019b9a0"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:07.563855 24552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:07.583452 24552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:07.586545 24552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:07.586746 24552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:07.587345 24552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:07.588258 24552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:07.588326 24552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:07.588389 24552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:07.588414 24552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:07.599241 24552 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:45673
I20260620 07:19:07.599272 24711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:45673 every 8 connection(s)
I20260620 07:19:07.600226 24552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:19:07.603556 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 24552
I20260620 07:19:07.604043 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.123.148:49705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:41153
--local_ip_for_outbound_sockets=127.11.123.130
--tserver_master_addrs=127.11.123.190:45703
--webserver_port=33155
--webserver_interface=127.11.123.130
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.123.148:49705/kms with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:19:07 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939947, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:07.622633 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:07.602423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:43907 (local address 127.11.123.190:45703)
0620 07:19:07.603052 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:07.603072 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:07.604364 (+ 1292us) server_negotiation.cc:407] Connection header received
0620 07:19:07.605970 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:07.605985 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:07.606402 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:07.606739 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:07.608422 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:07.610135 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:07.612014 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:07.612824 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:07.613017 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:07.616328 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:07.616336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:07.616356 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:07.616394 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:07.618445 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:07.619076 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:07.619083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:07.619089 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:07.619222 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:07.619573 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:07.619577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:07.619579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:07.620024 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:07.620144 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:07.621356 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0620 07:19:07.621636 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":127,"threads_started":1}
I20260620 07:19:07.623904 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:07.602569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45703 (local address 127.11.123.129:43907)
0620 07:19:07.603724 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:07.603777 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:07.605377 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:07.607247 (+ 1870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:07.607258 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:07.607596 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:07.608230 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:07.608253 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:07.610302 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:07.610309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:07.611846 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:07.611861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:07.613276 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:07.613284 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:07.613503 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:07.614245 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:07.614271 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:07.616097 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:07.618591 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:07.618597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:07.618612 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:07.618933 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:07.619321 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:07.619324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:07.619327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:07.619474 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:07.620196 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:07.620202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:07.620469 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 07:19:07.620657 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 07:19:07.620984 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":114,"threads_started":1}
I20260620 07:19:07.627170 24712 heartbeater.cc:347] Connected to a master server at 127.11.123.190:45703
I20260620 07:19:07.627414 24712 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:07.627930 24712 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
W20260620 07:19:07.630231 24712 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:45703 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:19:07.630343 24712 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:07.630520 24712 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
I20260620 07:19:07.631388 24712 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:07.631551 24712 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
W20260620 07:19:07.634466 24712 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:19:07.740767 24716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:07.741056 24716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:07.741168 24716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:07.745419 24716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:07.745494 24716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:07.745602 24716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:19:07.751188 24716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.123.148:49705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:41153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=33155
--tserver_master_addrs=127.11.123.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:07.752278 24716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:07.753307 24716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:07.763795 24721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:07.763808 24722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:07.764248 24716 server_base.cc:1034] running on GCE node
W20260620 07:19:07.763806 24724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:07.764757 24716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:07.765498 24716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:07.766691 24716 hybrid_clock.cc:648] HybridClock initialized: now 1781939947766669 us; error 39 us; skew 500 ppm
Jun 20 07:19:07 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939947, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:07.769747 24716 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:19:07.770740 24716 webserver.cc:460] Webserver started at http://127.11.123.130:33155/ using document root <none> and password file <none>
I20260620 07:19:07.771374 24716 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:07.771471 24716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:19:07 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939947, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:19:07.798045 24716 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.000s
I20260620 07:19:07.801261 24732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:07.802768 24716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:19:07.802888 24716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "84add220926047f68faf83b37de8372d"
format_stamp: "Formatted at 2026-06-20 07:19:06 on dist-test-slave-j663"
server_key: "14220e5a8c9fae697db577d7bbc42c71"
server_key_iv: "3940105dd825e6d37f542bf940315107"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:07.803565 24716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:07.823808 24716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:07.827315 24716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:07.827504 24716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:07.828159 24716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:07.829157 24716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:07.829228 24716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:07.829284 24716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:07.829314 24716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:07.841120 24716 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:41153
I20260620 07:19:07.841146 24845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:41153 every 8 connection(s)
I20260620 07:19:07.842079 24716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:19:07.851917 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 24716
I20260620 07:19:07.852455 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.123.148:49705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:44783
--local_ip_for_outbound_sockets=127.11.123.131
--tserver_master_addrs=127.11.123.190:45703
--webserver_port=41447
--webserver_interface=127.11.123.131
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.123.148:49705/kms with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:19:07 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939947, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:07.859483 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:07.844471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45703 (local address 127.11.123.130:45525)
0620 07:19:07.845607 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:07.845665 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:07.846662 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:07.847435 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:07.847444 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:07.847845 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:07.848480 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:07.848498 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:07.849672 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:07.849676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:07.851062 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:07.851073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:07.851712 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:07.851717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:07.851912 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:07.852577 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:07.852598 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:07.854312 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:07.856431 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:07.856437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:07.856452 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:07.856771 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:07.857109 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:07.857112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:07.857114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:07.857278 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:07.857757 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:07.857763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:07.858036 (+ 273us) client_negotiation.cc:769] Sending connection context
0620 07:19:07.858226 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 07:19:07.858534 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":121,"threads_started":1}
I20260620 07:19:07.860535 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:07.843978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:45525 (local address 127.11.123.190:45703)
0620 07:19:07.844138 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:07.844143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:07.845773 (+ 1630us) server_negotiation.cc:407] Connection header received
0620 07:19:07.846908 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:07.846912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:07.846973 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:07.847027 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:07.848665 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:07.849525 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:07.851198 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:07.851541 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:07.851665 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:07.854504 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:07.854509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:07.854512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:07.854544 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:07.856307 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:07.856888 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:07.856891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:07.856893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:07.856976 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:07.857387 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:07.857391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:07.857393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:07.857617 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:07.857706 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:07.860257 (+ 2551us) server_negotiation.cc:299] Negotiation successful
0620 07:19:07.860387 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:19:07.861621 24846 heartbeater.cc:347] Connected to a master server at 127.11.123.190:45703
I20260620 07:19:07.861845 24846 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:07.862383 24846 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
W20260620 07:19:07.864001 24846 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:45703 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:19:07.864136 24846 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:07.864311 24846 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
I20260620 07:19:07.865177 24846 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:07.865360 24846 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
W20260620 07:19:07.866143 24846 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:19:08.004578 24849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:08.004851 24849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:08.004937 24849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:08.009570 24849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:08.009642 24849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:08.009727 24849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:19:08.015367 24849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.123.148:49705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:44783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=41447
--tserver_master_addrs=127.11.123.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:08.016410 24849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:08.017853 24849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:08.029608 24856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:08.029618 24858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:08.030081 24855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:08.030355 24849 server_base.cc:1034] running on GCE node
I20260620 07:19:08.030791 24849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:08.031558 24849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:08.032758 24849 hybrid_clock.cc:648] HybridClock initialized: now 1781939948032738 us; error 33 us; skew 500 ppm
Jun 20 07:19:08 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:08.036284 24849 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:19:08.037328 24849 webserver.cc:460] Webserver started at http://127.11.123.131:41447/ using document root <none> and password file <none>
I20260620 07:19:08.038056 24849 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:08.038208 24849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:19:08 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:19:08.061036 24849 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 07:19:08.064240 24866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:08.066095 24849 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 07:19:08.066231 24849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "f591e9c45cf34128baf171a5bf07f913"
format_stamp: "Formatted at 2026-06-20 07:19:06 on dist-test-slave-j663"
server_key: "24c547e24bff949d0d2af4e4a889757a"
server_key_iv: "8a944ee507bfd3371781d3dfd6753679"
server_key_version: "kuduclusterkey@0"
I20260620 07:19:08.066974 24849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:08.083139 24849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:08.086603 24849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:08.086819 24849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:08.087525 24849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:08.088507 24849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:08.088581 24849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:08.088652 24849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:08.088723 24849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:08.100509 24849 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:44783
I20260620 07:19:08.100548 24979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:44783 every 8 connection(s)
I20260620 07:19:08.101723 24849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:19:08.111595 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 24849
Jun 20 07:19:08 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:08.119251 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:08.105230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:36477 (local address 127.11.123.190:45703)
0620 07:19:08.105360 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:08.105365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:08.105381 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:19:08.106074 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:08.106079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:08.106141 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:08.106201 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:08.107913 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:08.108710 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:08.110647 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:08.111014 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:08.111103 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:08.115062 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:08.115067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:08.115070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:08.115100 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:08.116707 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:08.117370 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:08.117375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:08.117376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:08.117462 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:08.117917 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:08.117923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:08.117925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:08.118177 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:08.118273 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:08.119014 (+ 741us) server_negotiation.cc:299] Negotiation successful
0620 07:19:08.119126 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:19:08.120388 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:08.104230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45703 (local address 127.11.123.131:36477)
0620 07:19:08.104703 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:08.104757 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:08.105778 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:08.106647 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:08.106657 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:08.107035 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:08.107734 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:08.107760 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:08.108841 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:08.108844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:08.110480 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:08.110493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:08.111322 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:08.111326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:08.111534 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:08.112942 (+ 1408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:08.112963 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:08.114836 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:08.116823 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:08.116831 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:08.116851 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:08.117261 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:08.117579 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:08.117582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:08.117584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:08.117786 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:08.118346 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:08.118353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:08.118711 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 07:19:08.118927 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 07:19:08.119283 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":132,"threads_started":1}
I20260620 07:19:08.121711 24980 heartbeater.cc:347] Connected to a master server at 127.11.123.190:45703
I20260620 07:19:08.121968 24980 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:08.122577 24980 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
W20260620 07:19:08.124331 24980 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:45703 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:19:08.124452 24980 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:08.124614 24980 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
I20260620 07:19:08.125802 24980 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:08.126015 24980 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
W20260620 07:19:08.126756 24980 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:19:08.128448 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:08.112674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.11.123.190:45703)
0620 07:19:08.116092 (+ 3418us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:08.116984 (+ 892us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:08.116996 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:19:08.117021 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:08.117023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:08.117059 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:08.117194 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:08.118352 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:08.118880 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:08.120255 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:08.120524 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:08.120624 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:08.125164 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:08.125168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:08.125170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:08.125196 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:08.126930 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:08.127319 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:08.127322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:08.127324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:08.127412 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:08.127634 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:08.127637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:08.127639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:08.127835 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:08.128037 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:08.128147 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 07:19:08.128275 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2981,"thread_start_us":71,"threads_started":1}
Jun 20 07:19:08 dist-test-slave-j663 krb5kdc[23518](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:08.634768 24712 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:08.635070 24712 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
I20260620 07:19:08.636214 24513 ts_manager.cc:194] Registered new tserver with Master: ddc9afc87dd34e0e8b3a1f8256e3dc8d (127.11.123.129:45673)
I20260620 07:19:08.639098 24513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:43907
Jun 20 07:19:08 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:19:08.866379 24846 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:08.866727 24846 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
I20260620 07:19:08.867563 24513 ts_manager.cc:194] Registered new tserver with Master: 84add220926047f68faf83b37de8372d (127.11.123.130:41153)
I20260620 07:19:08.868398 24513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:45525
I20260620 07:19:09.127013 24980 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:09.127306 24980 heartbeater.cc:511] Master 127.11.123.190:45703 requested a full tablet report, sending...
I20260620 07:19:09.128074 24513 ts_manager.cc:194] Registered new tserver with Master: f591e9c45cf34128baf171a5bf07f913 (127.11.123.131:44783)
I20260620 07:19:09.128880 24513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:36477
I20260620 07:19:09.137305 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:19:09.151916 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.138896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.11.123.190:45703)
0620 07:19:09.139141 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:09.139146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:09.139278 (+ 132us) server_negotiation.cc:407] Connection header received
0620 07:19:09.139441 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:09.139444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:09.139490 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:09.139592 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:09.140314 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.140853 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.142009 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.142398 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.142503 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:09.144232 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:09.144236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:09.144239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:09.144264 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:09.149290 (+ 5026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:09.149853 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:09.149856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:09.149858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:09.149959 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:09.150429 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:09.150433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:09.150434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:09.150841 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:09.150988 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:09.151675 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 07:19:09.151786 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
Jun 20 07:19:09 dist-test-slave-j663 krb5kdc[23518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for HTTP/127.11.123.148@KRBTEST.COM
I20260620 07:19:09.155563 24513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:19:09.159005 24513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:19:09.181301 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.173960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54548 (local address 127.11.123.129:45673)
0620 07:19:09.174454 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:09.174461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:09.175705 (+ 1244us) server_negotiation.cc:407] Connection header received
0620 07:19:09.175788 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:09.175794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:09.175997 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:09.176144 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:09.176450 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.177985 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.179601 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.180397 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.180487 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:09.180611 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:09.181021 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 07:19:09.181154 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":79,"threads_started":1}
I20260620 07:19:09.181946 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.173800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45673 (local address 127.0.0.1:54548)
0620 07:19:09.175490 (+ 1690us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:09.175503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:09.175604 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:09.176160 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:09.176165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:09.176190 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:09.176316 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:09.176322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.178111 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.178114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:09.179461 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:09.179471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.180704 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.180707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:09.180816 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:09.180854 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 07:19:09.181680 (+ 826us) client_negotiation.cc:240] Negotiation successful
0620 07:19:09.181808 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1432,"thread_start_us":75,"threads_started":1}
I20260620 07:19:09.185534 24645 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfcfa2cd5354320a14c07766da814ec (DEFAULT_TABLE table=test-table [id=7404ab7e011a4ca69fc76ee5bb05ed2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:09.186798 24645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfcfa2cd5354320a14c07766da814ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:09.187337 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.174078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:41153 (local address 127.0.0.1:55552)
0620 07:19:09.182444 (+ 8366us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:09.182457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:09.182526 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:09.183210 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:09.183213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:09.183228 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:09.183297 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:09.183301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.184621 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.184624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:09.185748 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:09.185760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.186974 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.186977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:09.187057 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:09.187078 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:19:09.187154 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 07:19:09.187216 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8313,"thread_start_us":81,"threads_started":1}
I20260620 07:19:09.187776 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.174429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.11.123.130:41153)
0620 07:19:09.176415 (+ 1986us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:09.176421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:09.182622 (+ 6201us) server_negotiation.cc:407] Connection header received
0620 07:19:09.182707 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:09.182712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:09.182916 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:09.183046 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:09.183409 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.184499 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.185884 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.186684 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.186800 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:09.186917 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:09.187556 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 07:19:09.187654 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1885,"thread_start_us":70,"threads_started":1}
I20260620 07:19:09.190584 24780 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfcfa2cd5354320a14c07766da814ec (DEFAULT_TABLE table=test-table [id=7404ab7e011a4ca69fc76ee5bb05ed2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:09.191668 24780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfcfa2cd5354320a14c07766da814ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:09.202363 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.172169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:44783 (local address 127.0.0.1:51118)
0620 07:19:09.174704 (+ 2535us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:09.174754 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:09.174907 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:09.192622 (+ 17715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:09.192625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:09.192646 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:09.192751 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:09.192757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.196524 (+ 3767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.196528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:09.197825 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:09.197837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.201933 (+ 4096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.201938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:09.202039 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:09.202068 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 07:19:09.202122 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 07:19:09.202203 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2358,"thread_start_us":98,"threads_started":1}
I20260620 07:19:09.203476 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.172905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.11.123.131:44783)
0620 07:19:09.189630 (+ 16725us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:09.189639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:09.189666 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:19:09.189741 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:09.189746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:09.189949 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:09.192915 (+ 2966us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:09.193101 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.194224 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.198844 (+ 4620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.199729 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.203032 (+ 3303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:09.203179 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:09.203232 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 07:19:09.203332 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16595,"thread_start_us":62,"threads_started":1}
I20260620 07:19:09.206120 25014 tablet_bootstrap.cc:492] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d: Bootstrap starting.
I20260620 07:19:09.206771 24910 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfcfa2cd5354320a14c07766da814ec (DEFAULT_TABLE table=test-table [id=7404ab7e011a4ca69fc76ee5bb05ed2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:09.207873 24910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfcfa2cd5354320a14c07766da814ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:09.208492 25014 tablet_bootstrap.cc:654] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:09.209779 25015 tablet_bootstrap.cc:492] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d: Bootstrap starting.
I20260620 07:19:09.210570 25014 log.cc:824] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:09.214461 25014 tablet_bootstrap.cc:492] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d: No bootstrap required, opened a new log
I20260620 07:19:09.214684 25014 ts_tablet_manager.cc:1397] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 07:19:09.215536 25015 tablet_bootstrap.cc:654] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:09.216280 25015 log.cc:824] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:09.219638 25015 tablet_bootstrap.cc:492] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d: No bootstrap required, opened a new log
I20260620 07:19:09.220029 25017 tablet_bootstrap.cc:492] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913: Bootstrap starting.
I20260620 07:19:09.221267 25015 ts_tablet_manager.cc:1397] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260620 07:19:09.222378 25014 raft_consensus.cc:348] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f591e9c45cf34128baf171a5bf07f913" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44783 } } peers { permanent_uuid: "84add220926047f68faf83b37de8372d" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 41153 } } peers { permanent_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45673 } }
I20260620 07:19:09.222666 25014 raft_consensus.cc:374] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:09.222728 25014 raft_consensus.cc:729] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84add220926047f68faf83b37de8372d, State: Initialized, Role: FOLLOWER
I20260620 07:19:09.223348 25014 consensus_queue.cc:260] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f591e9c45cf34128baf171a5bf07f913" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44783 } } peers { permanent_uuid: "84add220926047f68faf83b37de8372d" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 41153 } } peers { permanent_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45673 } }
I20260620 07:19:09.224301 25017 tablet_bootstrap.cc:654] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:09.224443 25014 ts_tablet_manager.cc:1428] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260620 07:19:09.224804 24846 heartbeater.cc:503] Master 127.11.123.190:45703 was elected leader, sending a full tablet report...
I20260620 07:19:09.227129 25015 raft_consensus.cc:348] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f591e9c45cf34128baf171a5bf07f913" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44783 } } peers { permanent_uuid: "84add220926047f68faf83b37de8372d" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 41153 } } peers { permanent_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45673 } }
I20260620 07:19:09.228272 25017 log.cc:824] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:09.228737 25015 raft_consensus.cc:374] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:09.228852 25015 raft_consensus.cc:729] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc9afc87dd34e0e8b3a1f8256e3dc8d, State: Initialized, Role: FOLLOWER
I20260620 07:19:09.229420 25015 consensus_queue.cc:260] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f591e9c45cf34128baf171a5bf07f913" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44783 } } peers { permanent_uuid: "84add220926047f68faf83b37de8372d" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 41153 } } peers { permanent_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45673 } }
I20260620 07:19:09.230633 25015 ts_tablet_manager.cc:1428] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260620 07:19:09.231026 24712 heartbeater.cc:503] Master 127.11.123.190:45703 was elected leader, sending a full tablet report...
I20260620 07:19:09.233263 25017 tablet_bootstrap.cc:492] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913: No bootstrap required, opened a new log
I20260620 07:19:09.233505 25017 ts_tablet_manager.cc:1397] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260620 07:19:09.236362 25017 raft_consensus.cc:348] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f591e9c45cf34128baf171a5bf07f913" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44783 } } peers { permanent_uuid: "84add220926047f68faf83b37de8372d" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 41153 } } peers { permanent_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45673 } }
I20260620 07:19:09.236613 25017 raft_consensus.cc:374] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:09.236678 25017 raft_consensus.cc:729] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f591e9c45cf34128baf171a5bf07f913, State: Initialized, Role: FOLLOWER
I20260620 07:19:09.237162 25017 consensus_queue.cc:260] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f591e9c45cf34128baf171a5bf07f913" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44783 } } peers { permanent_uuid: "84add220926047f68faf83b37de8372d" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 41153 } } peers { permanent_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45673 } }
I20260620 07:19:09.238368 25017 ts_tablet_manager.cc:1428] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:19:09.238711 24980 heartbeater.cc:503] Master 127.11.123.190:45703 was elected leader, sending a full tablet report...
W20260620 07:19:09.344918 24847 tablet.cc:2369] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:19:09.353050 24713 tablet.cc:2369] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:19:09.353403 24981 tablet.cc:2369] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:19:09.417194 25021 raft_consensus.cc:482] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:19:09.417426 25021 raft_consensus.cc:504] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f591e9c45cf34128baf171a5bf07f913" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44783 } } peers { permanent_uuid: "84add220926047f68faf83b37de8372d" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 41153 } } peers { permanent_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45673 } }
I20260620 07:19:09.418783 25021 leader_election.cc:288] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f591e9c45cf34128baf171a5bf07f913 (127.11.123.131:44783), 84add220926047f68faf83b37de8372d (127.11.123.130:41153)
I20260620 07:19:09.424731 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.420063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:56313 (local address 127.11.123.131:44783)
0620 07:19:09.420358 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:09.420362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:09.420380 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:19:09.420434 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:09.420437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:09.420482 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:09.420560 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:09.420991 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.421784 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.423360 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.423981 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.424109 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:09.424163 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:09.424500 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 07:19:09.424587 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260620 07:19:09.425340 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.419198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:44783 (local address 127.11.123.129:56313)
0620 07:19:09.420204 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:09.420219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:09.420323 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:09.420688 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:09.420691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:09.420706 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:09.420863 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:09.420886 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.421916 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.421920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:09.423208 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:09.423218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.424227 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.424231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:09.424349 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:09.424373 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:19:09.425096 (+ 723us) client_negotiation.cc:240] Negotiation successful
0620 07:19:09.425179 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":121,"threads_started":1}
I20260620 07:19:09.426106 24922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfcfa2cd5354320a14c07766da814ec" candidate_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f591e9c45cf34128baf171a5bf07f913" is_pre_election: true
I20260620 07:19:09.426426 24922 raft_consensus.cc:2454] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddc9afc87dd34e0e8b3a1f8256e3dc8d in term 0.
I20260620 07:19:09.426927 24599 leader_election.cc:302] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddc9afc87dd34e0e8b3a1f8256e3dc8d, f591e9c45cf34128baf171a5bf07f913; no voters:
I20260620 07:19:09.427201 25021 raft_consensus.cc:2781] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:19:09.427281 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.420233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:40429 (local address 127.11.123.130:41153)
0620 07:19:09.420719 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:09.420724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:09.420736 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:19:09.420781 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:09.420784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:09.420823 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:09.420889 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:09.423567 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.424319 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.425857 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.426452 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.426536 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:09.426578 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:09.427098 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 07:19:09.427171 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409}
I20260620 07:19:09.427287 25021 raft_consensus.cc:482] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:19:09.427336 25021 raft_consensus.cc:3037] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:09.427768 25024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.419581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:41153 (local address 127.11.123.129:40429)
0620 07:19:09.420498 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:09.420510 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:09.420578 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:09.423343 (+ 2765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:09.423345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:09.423365 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:09.423451 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:09.423456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.424463 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.424466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:09.425725 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:09.425735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.426667 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.426671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:09.426763 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:09.426782 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 07:19:09.427566 (+ 784us) client_negotiation.cc:240] Negotiation successful
0620 07:19:09.427635 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":90,"threads_started":1}
I20260620 07:19:09.428303 24800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfcfa2cd5354320a14c07766da814ec" candidate_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84add220926047f68faf83b37de8372d" is_pre_election: true
I20260620 07:19:09.428555 24800 raft_consensus.cc:2454] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddc9afc87dd34e0e8b3a1f8256e3dc8d in term 0.
I20260620 07:19:09.428561 25021 raft_consensus.cc:504] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f591e9c45cf34128baf171a5bf07f913" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44783 } } peers { permanent_uuid: "84add220926047f68faf83b37de8372d" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 41153 } } peers { permanent_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45673 } }
I20260620 07:19:09.428916 25021 leader_election.cc:288] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [CANDIDATE]: Term 1 election: Requested vote from peers f591e9c45cf34128baf171a5bf07f913 (127.11.123.131:44783), 84add220926047f68faf83b37de8372d (127.11.123.130:41153)
I20260620 07:19:09.429469 24922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfcfa2cd5354320a14c07766da814ec" candidate_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f591e9c45cf34128baf171a5bf07f913"
I20260620 07:19:09.429576 24800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfcfa2cd5354320a14c07766da814ec" candidate_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84add220926047f68faf83b37de8372d"
I20260620 07:19:09.429585 24922 raft_consensus.cc:3037] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:09.429697 24800 raft_consensus.cc:3037] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:09.430827 24800 raft_consensus.cc:2454] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddc9afc87dd34e0e8b3a1f8256e3dc8d in term 1.
I20260620 07:19:09.430827 24922 raft_consensus.cc:2454] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddc9afc87dd34e0e8b3a1f8256e3dc8d in term 1.
I20260620 07:19:09.431121 24599 leader_election.cc:302] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84add220926047f68faf83b37de8372d, ddc9afc87dd34e0e8b3a1f8256e3dc8d; no voters:
I20260620 07:19:09.431279 25021 raft_consensus.cc:2781] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:19:09.431493 25021 raft_consensus.cc:686] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [term 1 LEADER]: Becoming Leader. State: Replica: ddc9afc87dd34e0e8b3a1f8256e3dc8d, State: Running, Role: LEADER
I20260620 07:19:09.431811 25021 consensus_queue.cc:237] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f591e9c45cf34128baf171a5bf07f913" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44783 } } peers { permanent_uuid: "84add220926047f68faf83b37de8372d" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 41153 } } peers { permanent_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45673 } }
I20260620 07:19:09.434818 24514 catalog_manager.cc:5495] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d reported cstate change: term changed from 0 to 1, leader changed from <none> to ddc9afc87dd34e0e8b3a1f8256e3dc8d (127.11.123.129). New cstate: current_term: 1 leader_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f591e9c45cf34128baf171a5bf07f913" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84add220926047f68faf83b37de8372d" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 41153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddc9afc87dd34e0e8b3a1f8256e3dc8d" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45673 } health_report { overall_health: HEALTHY } } }
I20260620 07:19:09.478379 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.474803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54560 (local address 127.11.123.129:45673)
0620 07:19:09.474927 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:09.474931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:09.474946 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:19:09.475066 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:09.475068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:09.475121 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:09.475232 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:19:09.475479 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.476043 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.476887 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:09.477137 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:09.477249 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:09.477553 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:19:09.477967 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:19:09.478098 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:09.478162 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 07:19:09.478216 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 07:19:09.484241 24922 raft_consensus.cc:1261] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913 [term 1 FOLLOWER]: Refusing update from remote peer ddc9afc87dd34e0e8b3a1f8256e3dc8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:19:09.484241 24800 raft_consensus.cc:1261] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d [term 1 FOLLOWER]: Refusing update from remote peer ddc9afc87dd34e0e8b3a1f8256e3dc8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:19:09.484978 25021 consensus_queue.cc:1035] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f591e9c45cf34128baf171a5bf07f913" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 44783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:19:09.485206 25028 consensus_queue.cc:1035] T 6bfcfa2cd5354320a14c07766da814ec P ddc9afc87dd34e0e8b3a1f8256e3dc8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84add220926047f68faf83b37de8372d" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 41153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:19:09.494472 25038 mvcc.cc:205] Tried to move back new op lower bound from 7298826033079566336 to 7298826032874860544. Current Snapshot: MvccSnapshot[applied={T|T < 7298826033079566336}]
I20260620 07:19:09.494853 25037 mvcc.cc:205] Tried to move back new op lower bound from 7298826033079566336 to 7298826032874860544. Current Snapshot: MvccSnapshot[applied={T|T < 7298826033079566336}]
I20260620 07:19:09.504408 24510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:19:09.504601 24510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:19:09.507328 24510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5877b27ebe22481b81c102718a43a6dc: Sending DeleteTablet for 3 replicas of tablet 6bfcfa2cd5354320a14c07766da814ec
I20260620 07:19:09.508157 24910 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bfcfa2cd5354320a14c07766da814ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:09 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:51118
I20260620 07:19:09.508726 25046 tablet_replica.cc:330] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913: stopping tablet replica
I20260620 07:19:09.508698 24780 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bfcfa2cd5354320a14c07766da814ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:09 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:55552
I20260620 07:19:09.509254 25046 raft_consensus.cc:2229] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:19:09.509639 25046 raft_consensus.cc:2258] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:19:09.510010 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 24552
I20260620 07:19:09.513542 25046 ts_tablet_manager.cc:1905] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:19:09.516147 25046 ts_tablet_manager.cc:1918] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:19:09.516238 25046 log.cc:1196] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/6bfcfa2cd5354320a14c07766da814ec
I20260620 07:19:09.516582 25046 ts_tablet_manager.cc:1939] T 6bfcfa2cd5354320a14c07766da814ec P f591e9c45cf34128baf171a5bf07f913: Deleting consensus metadata
W20260620 07:19:09.517724 24501 connection.cc:535] client connection to 127.11.123.129:45673 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 07:19:09.517899 24501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 07:19:09.517966 24498 catalog_manager.cc:4841] TS f591e9c45cf34128baf171a5bf07f913 (127.11.123.131:44783): tablet 6bfcfa2cd5354320a14c07766da814ec (table test-table [id=7404ab7e011a4ca69fc76ee5bb05ed2f]) successfully deleted
I20260620 07:19:09.518661 25047 tablet_replica.cc:330] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d: stopping tablet replica
I20260620 07:19:09.518898 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.518625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45673 (local address 127.0.0.1:54564)
0620 07:19:09.518753 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:09.518819 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:45673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 07:19:09.518983 25047 raft_consensus.cc:2229] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 07:19:09.519111 24501 catalog_manager.cc:4584] TS ddc9afc87dd34e0e8b3a1f8256e3dc8d (127.11.123.129:45673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6bfcfa2cd5354320a14c07766da814ec: Network error: Client connection negotiation failed: client connection to 127.11.123.129:45673: connect: Connection refused (error 111)
I20260620 07:19:09.519308 25047 raft_consensus.cc:2258] T 6bfcfa2cd5354320a14c07766da814ec P 84add220926047f68faf83b37de8372d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:19:09.519706 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 24716
I20260620 07:19:09.525442 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 24849
I20260620 07:19:09.527464 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:09.525478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:41153 (local address 127.0.0.1:55554)
0620 07:19:09.527293 (+ 1815us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:09.527343 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.130:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1739}
I20260620 07:19:09.532343 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 24481
I20260620 07:19:09.539075 11758 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 07:19:09.883963 11758 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 07:19:09.884069 11758 mini_ranger.cc:66] Stopping Ranger...
I20260620 07:19:10.228621 11758 mini_ranger.cc:68] Stopped Ranger
2026-06-20 07:19:10.228 UTC [23548] LOG: received smart shutdown request
2026-06-20 07:19:10.232 UTC [23548] LOG: background worker "logical replication launcher" (PID 23557) exited with exit code 1
2026-06-20 07:19:10.232 UTC [23552] LOG: shutting down
2026-06-20 07:19:10.395 UTC [23548] LOG: database system is shut down
2026-06-20T07:19:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48718 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:19:10 dist-test-slave-j663 krb5kdc[25057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:19:10 dist-test-slave-j663 krb5kdc[25057](info): set up 2 sockets
Jun 20 07:19:10 dist-test-slave-j663 krb5kdc[25057](info): commencing operation
krb5kdc: starting...
W20260620 07:19:11.553007 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:19:11 dist-test-slave-j663 krb5kdc[25057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:19:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:19:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:19:11.668324 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:35859
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:35859,127.11.123.189:45573,127.11.123.188:45389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:11.776799 25070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:11.777045 25070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:11.777104 25070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:11.781091 25070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:19:11.781148 25070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:11.781168 25070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:11.781184 25070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:19:11.781232 25070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:19:11.786446 25070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:35859,127.11.123.189:45573,127.11.123.188:45389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:35859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:11.787518 25070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:11.788520 25070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:11.796934 25078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:11.796936 25076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:11.797122 25070 server_base.cc:1034] running on GCE node
W20260620 07:19:11.796936 25075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:11.797811 25070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:11.798847 25070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:11.800004 25070 hybrid_clock.cc:648] HybridClock initialized: now 1781939951799971 us; error 46 us; skew 500 ppm
Jun 20 07:19:11 dist-test-slave-j663 krb5kdc[25057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939951, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:11.802626 25070 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:19:11.803401 25070 webserver.cc:460] Webserver started at http://127.11.123.190:44701/ using document root <none> and password file <none>
I20260620 07:19:11.803895 25070 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:11.803953 25070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:11.804105 25070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:19:11.806087 25070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "88cd60dea76f42efa46cb946e957da83"
format_stamp: "Formatted at 2026-06-20 07:19:11 on dist-test-slave-j663"
server_key: "774a98ac0061ea676e5989572d2ae55d"
server_key_iv: "4699cc4ca994785cd8bb7042f1d36c53"
server_key_version: "encryptionkey@0"
I20260620 07:19:11.806572 25070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "88cd60dea76f42efa46cb946e957da83"
format_stamp: "Formatted at 2026-06-20 07:19:11 on dist-test-slave-j663"
server_key: "774a98ac0061ea676e5989572d2ae55d"
server_key_iv: "4699cc4ca994785cd8bb7042f1d36c53"
server_key_version: "encryptionkey@0"
I20260620 07:19:11.810036 25070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:19:11.812256 25085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:11.813580 25070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:19:11.813690 25070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "88cd60dea76f42efa46cb946e957da83"
format_stamp: "Formatted at 2026-06-20 07:19:11 on dist-test-slave-j663"
server_key: "774a98ac0061ea676e5989572d2ae55d"
server_key_iv: "4699cc4ca994785cd8bb7042f1d36c53"
server_key_version: "encryptionkey@0"
I20260620 07:19:11.813773 25070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:11.830825 25070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:11.833505 25070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:11.833660 25070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:11.840152 25070 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:35859
I20260620 07:19:11.840427 25137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:35859 every 8 connection(s)
I20260620 07:19:11.841111 25070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:19:11.843747 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 25070
I20260620 07:19:11.843868 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:19:11.843946 25138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:11.844167 11758 external_mini_cluster.cc:1428] Setting key 5d60b2862a4bc04d4473a37d0700cf77
I20260620 07:19:11.847939 25138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.123.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.189@KRBTEST.COM" created.
Jun 20 07:19:11 dist-test-slave-j663 krb5kdc[25057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939951, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:11.862986 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:11.850087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:35859 (local address 127.0.0.1:56018)
0620 07:19:11.850540 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:11.850589 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:11.851387 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:11.852175 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:11.852180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:11.852474 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:11.852712 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:11.852724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:11.853949 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:11.853952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:11.854899 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:11.854909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:11.855572 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:11.855578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:11.855740 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:11.856433 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:11.856449 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:11.857853 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:11.860181 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:11.860189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:11.860196 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:11.860415 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:11.860798 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:11.860800 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:11.860802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:11.860903 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:11.861659 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:11.861665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:11.861783 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 07:19:11.861928 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 07:19:11.862168 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":138,"threads_started":1}
I20260620 07:19:11.862991 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:11.850088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.11.123.190:35859)
0620 07:19:11.850640 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:11.850652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:11.850696 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:19:11.851700 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:11.851715 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:11.851966 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:11.852172 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:11.852901 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:11.853803 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:11.855021 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:11.855395 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:11.855527 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:11.858007 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:11.858016 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:11.858032 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:11.858066 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:11.860050 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:11.860530 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:11.860537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:11.860540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:11.860691 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:11.860998 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:11.861002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:11.861003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:11.861524 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:11.861688 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:11.862042 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 07:19:11.862303 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":435,"spinlock_wait_cycles":5888,"thread_start_us":165,"threads_started":1}
I20260620 07:19:11.865417 25138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:19:11.866129 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:11.865847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:45573 (local address 127.0.0.1:37832)
0620 07:19:11.865971 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:11.866073 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:45573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 07:19:11.866312 25087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.123.189:45573: connect: Connection refused (error 111)
I20260620 07:19:11.867195 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:11.866996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:45573 (local address 127.0.0.1:37842)
0620 07:19:11.867094 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:11.867147 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:45573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 07:19:11.867336 25138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.189:45573: Network error: Client connection negotiation failed: client connection to 127.11.123.189:45573: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.189.keytab.
Entry for principal kudu/127.11.123.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.189.keytab.
I20260620 07:19:11.871836 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.189:45573
--webserver_interface=127.11.123.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.189.keytab
--principal=kudu/127.11.123.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:35859,127.11.123.189:45573,127.11.123.188:45389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:19:11.916623 25138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } attempt: 1
I20260620 07:19:11.917573 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:11.917298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:45573 (local address 127.0.0.1:37848)
0620 07:19:11.917423 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:11.917493 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:45573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 07:19:11.918354 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:11.918123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:45573 (local address 127.0.0.1:37854)
0620 07:19:11.918234 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:11.918286 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:45573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 07:19:11.918501 25138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.189:45573: Network error: Client connection negotiation failed: client connection to 127.11.123.189:45573: connect: Connection refused (error 111)
W20260620 07:19:11.980394 25145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:11.980633 25145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:11.980674 25145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:11.984660 25145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:19:11.984711 25145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:11.984730 25145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:11.984747 25145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:19:11.984762 25145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:19:11.986689 25138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } attempt: 2
I20260620 07:19:11.987496 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:11.987214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:45573 (local address 127.0.0.1:37868)
0620 07:19:11.987359 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:11.987424 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:45573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 07:19:11.988209 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:11.988000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:45573 (local address 127.0.0.1:37870)
0620 07:19:11.988103 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:11.988148 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:45573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:19:11.988369 25138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.189:45573: Network error: Client connection negotiation failed: client connection to 127.11.123.189:45573: connect: Connection refused (error 111)
I20260620 07:19:11.990057 25145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:35859,127.11.123.189:45573,127.11.123.188:45389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.189.keytab
--principal=kudu/127.11.123.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.189:45573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.123.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:11.990957 25145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:11.991864 25145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:11.999684 25150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:11.999703 25151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:11.999691 25153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:12.000224 25145 server_base.cc:1034] running on GCE node
I20260620 07:19:12.000762 25145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:12.001880 25145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:12.003069 25145 hybrid_clock.cc:648] HybridClock initialized: now 1781939952003055 us; error 28 us; skew 500 ppm
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939952, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:12.005955 25145 init.cc:377] Logged in from keytab as kudu/127.11.123.189@KRBTEST.COM (short username kudu)
I20260620 07:19:12.006801 25145 webserver.cc:460] Webserver started at http://127.11.123.189:37829/ using document root <none> and password file <none>
I20260620 07:19:12.007381 25145 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:12.007440 25145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:12.007645 25145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:19:12.009646 25145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/data/instance:
uuid: "febaf1d70535497dba54f5aa1874ee3f"
format_stamp: "Formatted at 2026-06-20 07:19:12 on dist-test-slave-j663"
server_key: "3f09796a0b8b7d7e00cc364b0c0c02ea"
server_key_iv: "d5de90486412c42983fd619427b11497"
server_key_version: "encryptionkey@0"
I20260620 07:19:12.010191 25145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/wal/instance:
uuid: "febaf1d70535497dba54f5aa1874ee3f"
format_stamp: "Formatted at 2026-06-20 07:19:12 on dist-test-slave-j663"
server_key: "3f09796a0b8b7d7e00cc364b0c0c02ea"
server_key_iv: "d5de90486412c42983fd619427b11497"
server_key_version: "encryptionkey@0"
I20260620 07:19:12.013953 25145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:19:12.016288 25160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:12.017627 25145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:19:12.017745 25145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/wal
uuid: "febaf1d70535497dba54f5aa1874ee3f"
format_stamp: "Formatted at 2026-06-20 07:19:12 on dist-test-slave-j663"
server_key: "3f09796a0b8b7d7e00cc364b0c0c02ea"
server_key_iv: "d5de90486412c42983fd619427b11497"
server_key_version: "encryptionkey@0"
I20260620 07:19:12.017841 25145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:12.034178 25145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:12.037403 25145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:12.037622 25145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:12.044588 25145 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.189:45573
I20260620 07:19:12.044610 25212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.189:45573 every 8 connection(s)
I20260620 07:19:12.045495 25145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
I20260620 07:19:12.047694 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 25145
I20260620 07:19:12.047821 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-1/wal/instance
I20260620 07:19:12.048084 11758 external_mini_cluster.cc:1428] Setting key 1523534021a157542ae61c61262628c0
I20260620 07:19:12.048226 25213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:12.051791 25213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.123.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.188@KRBTEST.COM" created.
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939952, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.189@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:12.066197 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.053627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.11.123.190:35859)
0620 07:19:12.053764 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.053767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.054695 (+ 928us) server_negotiation.cc:407] Connection header received
0620 07:19:12.055565 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.055568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.055633 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.055732 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.056870 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.057471 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.059034 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.059268 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.059393 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.062037 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.062042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.062045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.062073 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.064132 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.064679 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.064684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.064686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.064767 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.065123 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.065126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.065128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.065302 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.065409 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.065895 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.066015 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:19:12.066903 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.054074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:35859 (local address 127.0.0.1:56024)
0620 07:19:12.054542 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.054592 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:12.055362 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:12.055894 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:12.055900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:12.056215 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:12.056732 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.056744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.057643 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.057651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.058900 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.058915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.059439 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.059444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.059613 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.060180 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:12.060195 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:12.061867 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:12.064269 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.064274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.064287 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.064534 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.064860 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.064865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.064866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.064998 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.065407 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:12.065414 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:12.065651 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 07:19:12.065828 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 07:19:12.066120 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
I20260620 07:19:12.068508 25213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.188.keytab.
Entry for principal kudu/127.11.123.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.188.keytab.
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939952, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.189@KRBTEST.COM for kudu/127.11.123.189@KRBTEST.COM
I20260620 07:19:12.074324 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.188:45389
--webserver_interface=127.11.123.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.188.keytab
--principal=kudu/127.11.123.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:35859,127.11.123.189:45573,127.11.123.188:45389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:19:12.078163 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.068936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:45573 (local address 127.0.0.1:37884)
0620 07:19:12.069060 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.069093 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:12.069168 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:12.069789 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:12.069792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:12.069917 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:12.069999 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.070004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.071078 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.071081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.071647 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.071655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.072281 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.072284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.072355 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.072742 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:12.072753 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:12.074175 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:12.076402 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.076405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.076406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.076629 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.076950 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.076953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.076954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.077041 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.077684 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:12.077687 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:12.077775 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 07:19:12.077857 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:19:12.077954 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 07:19:12.078454 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.069119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.11.123.189:45573)
0620 07:19:12.069375 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.069385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.069426 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:19:12.069473 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.069481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.069676 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.069785 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.070097 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.070901 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.071749 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.072178 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.072266 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.074306 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.074314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.074323 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.074343 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.076285 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.076742 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.076750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.076754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.076847 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.077207 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.077213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.077218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.077549 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.077694 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.078003 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.078168 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
I20260620 07:19:12.079520 25138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } attempt: 3
I20260620 07:19:12.080310 25213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:19:12.081038 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.080738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34384)
0620 07:19:12.080905 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.080976 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260620 07:19:12.081248 25162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
I20260620 07:19:12.082187 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.081922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34392)
0620 07:19:12.082049 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.082111 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 07:19:12.082942 25213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:45389: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939951, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.189@KRBTEST.COM
I20260620 07:19:12.088023 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.080016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.11.123.189:45573)
0620 07:19:12.080507 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.080514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.080528 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:19:12.080566 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.080572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.080622 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.080710 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.081253 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.081954 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.082847 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.083212 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.083321 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.085200 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.085203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.085206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.085228 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.086489 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.086903 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.086909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.086912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.086987 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.087262 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.087267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.087268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.087427 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.087529 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.087793 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.087892 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401}
I20260620 07:19:12.088025 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.079985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:45573 (local address 127.0.0.1:37896)
0620 07:19:12.080210 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.080226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:12.080333 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:12.080754 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:12.080760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:12.080944 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:12.081110 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.081121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.082107 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.082110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.082726 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.082736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.083325 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.083328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.083442 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.083889 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:12.083898 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:12.085025 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:12.086608 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.086613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.086614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.086805 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.087083 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.087086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.087088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.087172 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.087521 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:12.087524 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:12.087620 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 07:19:12.087734 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 07:19:12.087871 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 07:19:12.089030 25138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:19:12.089685 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.089468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34398)
0620 07:19:12.089574 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.089622 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 07:19:12.090293 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.090073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34412)
0620 07:19:12.090198 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.090241 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 07:19:12.090431 25138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:45389: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
I20260620 07:19:12.121608 25138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } attempt: 1
I20260620 07:19:12.122452 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.122111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34418)
0620 07:19:12.122311 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.122379 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 07:19:12.123153 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.122943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34432)
0620 07:19:12.123056 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.123104 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:19:12.123286 25138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:45389: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
I20260620 07:19:12.132220 25213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } attempt: 1
I20260620 07:19:12.132949 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.132685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34434)
0620 07:19:12.132827 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.132883 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 07:19:12.133694 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.133508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34440)
0620 07:19:12.133602 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.133644 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 07:19:12.133893 25213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:45389: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
W20260620 07:19:12.184271 25219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:12.184508 25219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:12.184547 25219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:12.188534 25219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:19:12.188591 25219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:12.188611 25219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:12.188627 25219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:19:12.188642 25219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:19:12.193768 25219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:35859,127.11.123.189:45573,127.11.123.188:45389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.188.keytab
--principal=kudu/127.11.123.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.188:45389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.123.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:12.194738 25219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:12.195716 25219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:19:12.198429 25138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } attempt: 2
I20260620 07:19:12.199205 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.198930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34448)
0620 07:19:12.199060 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.199131 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 07:19:12.200052 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.199800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34458)
0620 07:19:12.199944 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.199991 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260620 07:19:12.200218 25138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:45389: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
I20260620 07:19:12.202069 25213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } attempt: 2
I20260620 07:19:12.202787 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.202486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34466)
0620 07:19:12.202654 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.202710 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 07:19:12.203742 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.203509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34470)
0620 07:19:12.203620 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.203677 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 07:19:12.203918 25213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:45389: Network error: Client connection negotiation failed: client connection to 127.11.123.188:45389: connect: Connection refused (error 111)
W20260620 07:19:12.204094 25225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:12.204164 25228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:12.204095 25226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:12.204674 25219 server_base.cc:1034] running on GCE node
I20260620 07:19:12.205194 25219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:12.206138 25219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:12.207321 25219 hybrid_clock.cc:648] HybridClock initialized: now 1781939952207305 us; error 29 us; skew 500 ppm
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939952, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:12.209966 25219 init.cc:377] Logged in from keytab as kudu/127.11.123.188@KRBTEST.COM (short username kudu)
I20260620 07:19:12.210754 25219 webserver.cc:460] Webserver started at http://127.11.123.188:42001/ using document root <none> and password file <none>
I20260620 07:19:12.211283 25219 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:12.211373 25219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:12.211596 25219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:19:12.213567 25219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/data/instance:
uuid: "529a5439509641b2a0e25c8a78b62437"
format_stamp: "Formatted at 2026-06-20 07:19:12 on dist-test-slave-j663"
server_key: "4f1d1f93166c4cb9b87a5167cbf7fa75"
server_key_iv: "bb1590281ab95e70e175d5f9eb61efeb"
server_key_version: "encryptionkey@0"
I20260620 07:19:12.214107 25219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/wal/instance:
uuid: "529a5439509641b2a0e25c8a78b62437"
format_stamp: "Formatted at 2026-06-20 07:19:12 on dist-test-slave-j663"
server_key: "4f1d1f93166c4cb9b87a5167cbf7fa75"
server_key_iv: "bb1590281ab95e70e175d5f9eb61efeb"
server_key_version: "encryptionkey@0"
I20260620 07:19:12.217537 25219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:19:12.219950 25235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:12.221163 25219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:19:12.221256 25219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/wal
uuid: "529a5439509641b2a0e25c8a78b62437"
format_stamp: "Formatted at 2026-06-20 07:19:12 on dist-test-slave-j663"
server_key: "4f1d1f93166c4cb9b87a5167cbf7fa75"
server_key_iv: "bb1590281ab95e70e175d5f9eb61efeb"
server_key_version: "encryptionkey@0"
I20260620 07:19:12.221333 25219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:12.256485 25219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:12.259689 25219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:12.259915 25219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:12.266613 25219 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.188:45389
I20260620 07:19:12.266909 25287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.188:45389 every 8 connection(s)
I20260620 07:19:12.267581 25219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
I20260620 07:19:12.270458 25288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:12.270840 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 25219
I20260620 07:19:12.270965 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939794025788-11758-0/minicluster-data/master-2/wal/instance
I20260620 07:19:12.271255 11758 external_mini_cluster.cc:1428] Setting key 653735b93c46669392507b4de1ddd05f
I20260620 07:19:12.275147 25288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.188@KRBTEST.COM
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939952, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.188@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:12.285256 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.272560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.11.123.188:45389)
0620 07:19:12.272954 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.272967 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.273000 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:19:12.273688 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.273704 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.274020 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.274283 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.275158 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.276154 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.277218 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.277696 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.277821 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.280324 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.280328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.280336 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.280360 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.282112 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.282573 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.282578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.282581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.282732 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.282999 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.283004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.283008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.283551 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.283742 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.284059 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.284323 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"spinlock_wait_cycles":7936,"thread_start_us":122,"threads_started":1}
I20260620 07:19:12.286439 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.276824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.11.123.190:35859)
0620 07:19:12.276941 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.276946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.277307 (+ 361us) server_negotiation.cc:407] Connection header received
0620 07:19:12.277464 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.277467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.277518 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.277607 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.278472 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.278948 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.279995 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.280341 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.280418 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.282781 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.282787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.282791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.282823 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.284411 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.284995 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.284999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.285001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.285109 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.285481 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.285485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.285486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.285690 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.285772 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.286168 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.286302 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 07:19:12.286477 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.276675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:35859 (local address 127.0.0.1:56038)
0620 07:19:12.277176 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.277233 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:12.277366 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:12.277655 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:12.277660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:12.278156 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:12.278342 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.278353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.279070 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.279074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.279869 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.279887 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.280445 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.280448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.280577 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.280997 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:12.281016 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:12.282651 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:12.284532 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.284542 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.284550 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.284877 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.285214 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.285218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.285222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.285351 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.285850 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:12.285855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:12.285958 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 07:19:12.286090 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 07:19:12.286249 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"spinlock_wait_cycles":3456,"thread_start_us":89,"threads_started":1}
I20260620 07:19:12.287699 25288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939952, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.188@KRBTEST.COM for kudu/127.11.123.189@KRBTEST.COM
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:12.295059 25213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } attempt: 3
I20260620 07:19:12.296399 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.288365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:45573 (local address 127.0.0.1:37906)
0620 07:19:12.288445 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.288458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:12.288536 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:12.288771 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:12.288774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:12.289008 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:12.289157 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.289165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.289863 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.289870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.290495 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.290505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.291161 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.291165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.291265 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.291682 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:12.291692 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:12.292846 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:12.294702 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.294706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.294710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.294942 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.295272 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.295278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.295281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.295384 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.295850 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:12.295853 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:12.295962 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 07:19:12.296102 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 07:19:12.296215 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 07:19:12.296487 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.288244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.11.123.189:45573)
0620 07:19:12.288356 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.288360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.288509 (+ 149us) server_negotiation.cc:407] Connection header received
0620 07:19:12.288613 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.288616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.288668 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.288760 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.289272 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.289755 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.290623 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.291026 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.291152 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.292974 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.292977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.292979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.293001 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.294586 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.295039 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.295041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.295042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.295159 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.295492 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.295496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.295498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.295709 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.295775 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.296187 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.296310 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 07:19:12.297062 25288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:19:12.298281 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.287586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.11.123.190:35859)
0620 07:19:12.287770 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.287774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.288154 (+ 380us) server_negotiation.cc:407] Connection header received
0620 07:19:12.288242 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.288248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.288294 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.288375 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.289300 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.290119 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.290993 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.291280 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.291399 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.293784 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.293787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.293790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.293812 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.295600 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.296055 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.296058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.296060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.296139 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.297040 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.297043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.297045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.297359 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.297579 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.298007 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.298116 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 07:19:12.299408 25138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } attempt: 3
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939952, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.189@KRBTEST.COM for kudu/127.11.123.188@KRBTEST.COM
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939952, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.188@KRBTEST.COM for kudu/127.11.123.188@KRBTEST.COM
I20260620 07:19:12.304073 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.295466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34492)
0620 07:19:12.295625 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.295637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:12.295710 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:12.295962 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:12.295965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:12.296138 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:12.296231 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.296238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.297118 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.297121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.297836 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.297847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.298378 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.298384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.298462 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.298923 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:12.298941 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:12.300397 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:12.302479 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.302482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.302484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.302725 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.303034 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.303037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.303039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.303123 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.303562 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:12.303568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:12.303667 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 07:19:12.303752 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:19:12.303883 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 07:19:12.304726 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.295536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.11.123.188:45389)
0620 07:19:12.295636 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.295643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.295681 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:19:12.295807 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.295811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.295850 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.295934 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.296353 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.297002 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.297958 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.298240 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.298661 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.300534 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.300537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.300540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.300561 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.302351 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.302847 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.302850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.302852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.302927 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.303236 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.303240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.303241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.303441 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.304367 (+ 926us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.304450 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.304558 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.189@KRBTEST.COM
I20260620 07:19:12.306735 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.297506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34500)
0620 07:19:12.297696 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.297708 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:12.297779 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:12.298236 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:12.298238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:12.298402 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:12.298475 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.298481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.299928 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.299931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.300663 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.300672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.301393 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.301396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.301463 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.301831 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:12.301845 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:12.303236 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:12.305363 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.305366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.305367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.305588 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.305886 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.305888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.305890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.305969 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.306333 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:12.306336 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:12.306400 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 07:19:12.306483 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:19:12.306588 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
Jun 20 07:19:12 dist-test-slave-j663 krb5kdc[25057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939951, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.188@KRBTEST.COM
I20260620 07:19:12.307170 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.297713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.11.123.188:45389)
0620 07:19:12.298018 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.298026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.298038 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:19:12.298070 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.298075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.298116 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.298203 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.298611 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.299801 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.300792 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.301184 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.301285 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.303350 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.303353 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.303355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.303373 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.305226 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.305695 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.305698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.305699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.305789 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.306060 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.306063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.306064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.306224 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.306303 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.306912 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.307020 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":141,"threads_started":1}
I20260620 07:19:12.308081 25213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f: Bootstrap starting.
I20260620 07:19:12.310128 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.299335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.11.123.189:45573)
0620 07:19:12.299430 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.299434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.300189 (+ 755us) server_negotiation.cc:407] Connection header received
0620 07:19:12.300358 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.300362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.300406 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.300454 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.301288 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.302039 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.303386 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.303799 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.303884 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.306929 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.306932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.306935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.306959 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.308517 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.308901 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.308904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.308906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.308987 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.309354 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.309361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.309364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.309535 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.309667 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.309908 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.310009 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 07:19:12.311096 25213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:12.311846 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.299742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:45389 (local address 127.0.0.1:34502)
0620 07:19:12.299899 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:12.299911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:12.299977 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:12.300579 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:12.300582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:12.301040 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:12.301154 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.301160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.302751 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.302754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.303532 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:12.303544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.305345 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.305349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:12.305469 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.305941 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:12.305955 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:12.307881 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:12.310335 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.310338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.310340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.310580 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.310898 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:12.310901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:12.310903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:12.310993 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:12.311442 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:12.311445 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:12.311525 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 07:19:12.311599 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 07:19:12.311709 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 07:19:12.311911 25213 log.cc:824] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:12.312474 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.299795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.11.123.188:45389)
0620 07:19:12.300354 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.300363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.300374 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:19:12.300403 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.300406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.300445 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.301280 (+ 835us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.301856 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.302623 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.303975 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.305183 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.308012 (+ 2829us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.308092 (+ 80us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.308094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.308096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.308107 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.310162 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.310698 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.310701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.310702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.310784 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.311107 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.311110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.311111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.311324 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.312134 (+ 810us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.312207 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.312330 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":98,"threads_started":1}
I20260620 07:19:12.313205 25288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437: Bootstrap starting.
I20260620 07:19:12.313959 25213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f: No bootstrap required, opened a new log
I20260620 07:19:12.315033 25138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83: Bootstrap starting.
I20260620 07:19:12.315562 25288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:12.316367 25288 log.cc:824] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:12.317163 25213 raft_consensus.cc:348] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } }
I20260620 07:19:12.317442 25213 raft_consensus.cc:374] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:12.317517 25213 raft_consensus.cc:729] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: febaf1d70535497dba54f5aa1874ee3f, State: Initialized, Role: FOLLOWER
I20260620 07:19:12.317852 25138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:12.317998 25213 consensus_queue.cc:260] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } }
I20260620 07:19:12.318470 25297 sys_catalog.cc:455] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } } }
I20260620 07:19:12.318631 25297 sys_catalog.cc:458] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:19:12.318696 25138 log.cc:824] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:12.319028 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:19:12.319355 25213 sys_catalog.cc:564] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:19:12.320760 25288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437: No bootstrap required, opened a new log
I20260620 07:19:12.323817 25138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83: No bootstrap required, opened a new log
W20260620 07:19:12.324389 25313 catalog_manager.cc:1558] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:19:12.324512 25313 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:19:12.326114 25288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } }
I20260620 07:19:12.326357 25288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:12.326409 25288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529a5439509641b2a0e25c8a78b62437, State: Initialized, Role: FOLLOWER
I20260620 07:19:12.322988 25145 master_runner.cc:186] Error getting master registration for 127.11.123.190:35859: OK, instance_id { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" instance_seqno: 1781939951838385 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:19:12.326875 25145 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:19:12.326884 25288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } }
I20260620 07:19:12.327827 25138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } }
I20260620 07:19:12.328022 25138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:12.328075 25138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88cd60dea76f42efa46cb946e957da83, State: Initialized, Role: FOLLOWER
I20260620 07:19:12.328143 25288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:19:12.328480 25138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } }
I20260620 07:19:12.329988 25138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:19:12.330511 25319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } } }
I20260620 07:19:12.330634 25319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:19:12.330878 25318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } } }
I20260620 07:19:12.331014 25318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:19:12.331540 25219 master_runner.cc:186] Error getting master registration for 127.11.123.190:35859: OK, instance_id { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" instance_seqno: 1781939951838385 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260620 07:19:12.335475 25091 tablet.cc:2369] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:19:12.338999 25341 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:19:12.339079 25341 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:19:12.341125 25070 master_runner.cc:263] No leader master found from master 88cd60dea76f42efa46cb946e957da83
I20260620 07:19:12.341185 25070 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:19:12.341372 25219 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 07:19:12.336313 25330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:19:12.341506 25330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:19:12.349375 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.322916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.11.123.190:35859)
0620 07:19:12.323076 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.323080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.331371 (+ 8291us) server_negotiation.cc:407] Connection header received
0620 07:19:12.333712 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.333716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.333766 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.333814 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.335464 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.336176 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.341591 (+ 5415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.341905 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.341982 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.344837 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.344841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.344844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.344871 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.346623 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.347341 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.347345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.347347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.347649 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.348404 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.348407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.348411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.348629 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.348729 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.349068 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.349207 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 07:19:12.352192 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.323151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.11.123.189:45573)
0620 07:19:12.323263 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.323267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.331602 (+ 8335us) server_negotiation.cc:407] Connection header received
0620 07:19:12.333484 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.333487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.333537 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.333589 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.337329 (+ 3740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.338022 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.340992 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.341368 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.341448 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.346723 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.346728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.346731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.346757 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.348747 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.350811 (+ 2064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.350815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.350817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.350898 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.351252 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.351256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.351264 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.351485 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.351610 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.351932 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.352046 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:19:12.352273 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:12.322687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.11.123.188:45389)
0620 07:19:12.322795 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:12.322800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:12.331837 (+ 9037us) server_negotiation.cc:407] Connection header received
0620 07:19:12.332242 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:12.332248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:12.332307 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:12.332359 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:12.339149 (+ 6790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.339882 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.342873 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:12.343256 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:12.343335 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:12.347111 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:12.347115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:12.347118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:12.347146 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:12.349577 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.350413 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.350417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.350418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.350502 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:12.351288 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:12.351292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:12.351295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:12.351526 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:12.351654 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:12.351992 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 07:19:12.352115 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 07:19:12.387053 25318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:19:12.387256 25318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } }
I20260620 07:19:12.388322 25318 leader_election.cc:288] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88cd60dea76f42efa46cb946e957da83 (127.11.123.190:35859), febaf1d70535497dba54f5aa1874ee3f (127.11.123.189:45573)
I20260620 07:19:12.389211 25112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "529a5439509641b2a0e25c8a78b62437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88cd60dea76f42efa46cb946e957da83" is_pre_election: true
I20260620 07:19:12.389525 25112 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529a5439509641b2a0e25c8a78b62437 in term 0.
I20260620 07:19:12.390290 25187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "529a5439509641b2a0e25c8a78b62437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "febaf1d70535497dba54f5aa1874ee3f" is_pre_election: true
I20260620 07:19:12.390559 25187 raft_consensus.cc:2454] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529a5439509641b2a0e25c8a78b62437 in term 0.
I20260620 07:19:12.391036 25237 leader_election.cc:302] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529a5439509641b2a0e25c8a78b62437, febaf1d70535497dba54f5aa1874ee3f; no voters:
I20260620 07:19:12.391614 25318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:19:12.391696 25318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:19:12.391737 25318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:12.393020 25318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } }
I20260620 07:19:12.393483 25318 leader_election.cc:288] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [CANDIDATE]: Term 1 election: Requested vote from peers 88cd60dea76f42efa46cb946e957da83 (127.11.123.190:35859), febaf1d70535497dba54f5aa1874ee3f (127.11.123.189:45573)
I20260620 07:19:12.393978 25112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "529a5439509641b2a0e25c8a78b62437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88cd60dea76f42efa46cb946e957da83"
I20260620 07:19:12.394098 25112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:12.395340 25112 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529a5439509641b2a0e25c8a78b62437 in term 1.
I20260620 07:19:12.395816 25187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "529a5439509641b2a0e25c8a78b62437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "febaf1d70535497dba54f5aa1874ee3f"
I20260620 07:19:12.395934 25187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:12.397167 25187 raft_consensus.cc:2454] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529a5439509641b2a0e25c8a78b62437 in term 1.
I20260620 07:19:12.397526 25237 leader_election.cc:302] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529a5439509641b2a0e25c8a78b62437, febaf1d70535497dba54f5aa1874ee3f; no voters:
I20260620 07:19:12.397815 25318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:19:12.398056 25318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [term 1 LEADER]: Becoming Leader. State: Replica: 529a5439509641b2a0e25c8a78b62437, State: Running, Role: LEADER
I20260620 07:19:12.398461 25318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } }
I20260620 07:19:12.400306 25318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 529a5439509641b2a0e25c8a78b62437. Latest consensus state: current_term: 1 leader_uuid: "529a5439509641b2a0e25c8a78b62437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } } }
I20260620 07:19:12.400492 25318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:12.404487 25346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:19:12.405499 25346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:19:12.411098 25187 raft_consensus.cc:1261] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [term 1 FOLLOWER]: Refusing update from remote peer 529a5439509641b2a0e25c8a78b62437: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:19:12.413813 25344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [LEADER]: Connected to new peer: Peer: permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:19:12.418097 25297 sys_catalog.cc:455] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 529a5439509641b2a0e25c8a78b62437. Latest consensus state: current_term: 1 leader_uuid: "529a5439509641b2a0e25c8a78b62437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } } }
I20260620 07:19:12.418241 25297 sys_catalog.cc:458] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:19:12.419337 25112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [term 1 FOLLOWER]: Refusing update from remote peer 529a5439509641b2a0e25c8a78b62437: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:19:12.419965 25344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:19:12.424193 25319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 529a5439509641b2a0e25c8a78b62437. Latest consensus state: current_term: 1 leader_uuid: "529a5439509641b2a0e25c8a78b62437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } } }
I20260620 07:19:12.424310 25319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:19:12.430706 25344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "529a5439509641b2a0e25c8a78b62437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } } }
I20260620 07:19:12.430827 25344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:12.431926 25319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "529a5439509641b2a0e25c8a78b62437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } } }
I20260620 07:19:12.432021 25319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:19:12.432412 25347 mvcc.cc:205] Tried to move back new op lower bound from 7298826045068255232 to 7298826045027581952. Current Snapshot: MvccSnapshot[applied={T|T < 7298826045068255232}]
I20260620 07:19:12.433954 25350 mvcc.cc:205] Tried to move back new op lower bound from 7298826045068255232 to 7298826045027581952. Current Snapshot: MvccSnapshot[applied={T|T < 7298826045068255232}]
I20260620 07:19:12.434670 25318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "529a5439509641b2a0e25c8a78b62437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } } }
I20260620 07:19:12.434777 25318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437 [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:12.435658 25297 sys_catalog.cc:455] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "529a5439509641b2a0e25c8a78b62437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88cd60dea76f42efa46cb946e957da83" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35859 } } peers { permanent_uuid: "febaf1d70535497dba54f5aa1874ee3f" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 45573 } } peers { permanent_uuid: "529a5439509641b2a0e25c8a78b62437" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 45389 } } }
I20260620 07:19:12.435772 25297 sys_catalog.cc:458] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:19:12.442883 25348 mvcc.cc:205] Tried to move back new op lower bound from 7298826045068255232 to 7298826045027581952. Current Snapshot: MvccSnapshot[applied={T|T < 7298826045068255232}]
I20260620 07:19:12.443358 25346 catalog_manager.cc:1347] Generated new cluster ID: 157e6ddfaa214d3cbb1bb6683ea738c8
I20260620 07:19:12.443413 25346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:19:12.483222 25346 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:19:12.484514 25346 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:19:12.505271 25346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 529a5439509641b2a0e25c8a78b62437: Generated new TSK 0
I20260620 07:19:12.506131 25346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:19:13.325970 25313 catalog_manager.cc:1259] Loaded cluster ID: 157e6ddfaa214d3cbb1bb6683ea738c8
I20260620 07:19:13.326056 25313 catalog_manager.cc:1552] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f: loading cluster ID for follower catalog manager: success
I20260620 07:19:13.327705 25313 catalog_manager.cc:1574] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f: acquiring CA information for follower catalog manager: success
I20260620 07:19:13.328904 25313 catalog_manager.cc:1602] T 00000000000000000000000000000000 P febaf1d70535497dba54f5aa1874ee3f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:19:13.340109 25341 catalog_manager.cc:1259] Loaded cluster ID: 157e6ddfaa214d3cbb1bb6683ea738c8
I20260620 07:19:13.340168 25341 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83: loading cluster ID for follower catalog manager: success
I20260620 07:19:13.341651 25341 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83: acquiring CA information for follower catalog manager: success
I20260620 07:19:13.342722 25341 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88cd60dea76f42efa46cb946e957da83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:19:13.573506 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 25070
I20260620 07:19:13.579046 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 25145
I20260620 07:19:13.584563 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 25219
2026-06-20T07:19:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3072 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:19:13 dist-test-slave-j663 krb5kdc[25360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:19:13 dist-test-slave-j663 krb5kdc[25360](info): set up 2 sockets
Jun 20 07:19:13 dist-test-slave-j663 krb5kdc[25360](info): commencing operation
krb5kdc: starting...
W20260620 07:19:14.617365 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:19:14 dist-test-slave-j663 krb5kdc[25360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:19:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:19:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:19:14.733973 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:35635
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:35635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:14.840209 25373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:14.840485 25373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:14.840559 25373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:14.844513 25373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:19:14.844597 25373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:14.844635 25373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:14.844673 25373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:19:14.844703 25373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:19:14.849752 25373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:35635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:35635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:14.850721 25373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:14.851680 25373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:14.859676 25379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:14.859674 25381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:14.859700 25378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:14.859934 25373 server_base.cc:1034] running on GCE node
I20260620 07:19:14.860577 25373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:14.861644 25373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:14.862815 25373 hybrid_clock.cc:648] HybridClock initialized: now 1781939954862784 us; error 39 us; skew 500 ppm
Jun 20 07:19:14 dist-test-slave-j663 krb5kdc[25360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939954, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:14.865439 25373 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:19:14.866235 25373 webserver.cc:460] Webserver started at http://127.11.123.190:36925/ using document root <none> and password file <none>
I20260620 07:19:14.866755 25373 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:14.866815 25373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:14.867039 25373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:19:14.869009 25373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "1b27a42683f44c8f882530d469ee4ed6"
format_stamp: "Formatted at 2026-06-20 07:19:14 on dist-test-slave-j663"
server_key: "afa314c6414dcb1a8bf4c6483b42ff7d"
server_key_iv: "7df1f45e5824383474497eeaf48a8b1d"
server_key_version: "encryptionkey@0"
I20260620 07:19:14.869618 25373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "1b27a42683f44c8f882530d469ee4ed6"
format_stamp: "Formatted at 2026-06-20 07:19:14 on dist-test-slave-j663"
server_key: "afa314c6414dcb1a8bf4c6483b42ff7d"
server_key_iv: "7df1f45e5824383474497eeaf48a8b1d"
server_key_version: "encryptionkey@0"
I20260620 07:19:14.873147 25373 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:19:14.875623 25388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:14.876739 25373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:14.876878 25373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "1b27a42683f44c8f882530d469ee4ed6"
format_stamp: "Formatted at 2026-06-20 07:19:14 on dist-test-slave-j663"
server_key: "afa314c6414dcb1a8bf4c6483b42ff7d"
server_key_iv: "7df1f45e5824383474497eeaf48a8b1d"
server_key_version: "encryptionkey@0"
I20260620 07:19:14.876988 25373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:14.890573 25373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:14.893200 25373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:14.893383 25373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:14.900048 25373 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:35635
I20260620 07:19:14.900059 25440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:35635 every 8 connection(s)
I20260620 07:19:14.900928 25373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:19:14.903672 25441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:14.908861 25441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6: Bootstrap starting.
I20260620 07:19:14.909257 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 25373
I20260620 07:19:14.909376 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:19:14.909658 11758 external_mini_cluster.cc:1428] Setting key 85893eec6b67e130a1deec621168d557
I20260620 07:19:14.911368 25441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:14.912063 25441 log.cc:824] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:14.914184 25441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6: No bootstrap required, opened a new log
I20260620 07:19:14.919361 25441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b27a42683f44c8f882530d469ee4ed6" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35635 } }
Jun 20 07:19:14 dist-test-slave-j663 krb5kdc[25360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:14.919663 25441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:14.919764 25441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b27a42683f44c8f882530d469ee4ed6, State: Initialized, Role: FOLLOWER
I20260620 07:19:14.920305 25441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b27a42683f44c8f882530d469ee4ed6" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35635 } }
I20260620 07:19:14.920519 25441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:19:14.920636 25441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:19:14.921108 25441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:14.922161 25441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b27a42683f44c8f882530d469ee4ed6" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35635 } }
I20260620 07:19:14.922561 25441 leader_election.cc:302] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b27a42683f44c8f882530d469ee4ed6; no voters:
I20260620 07:19:14.922906 25441 leader_election.cc:288] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:19:14.923310 25447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:19:14.923578 25447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [term 1 LEADER]: Becoming Leader. State: Replica: 1b27a42683f44c8f882530d469ee4ed6, State: Running, Role: LEADER
I20260620 07:19:14.923952 25447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b27a42683f44c8f882530d469ee4ed6" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35635 } }
I20260620 07:19:14.924226 25441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:19:14.925261 25447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b27a42683f44c8f882530d469ee4ed6. Latest consensus state: current_term: 1 leader_uuid: "1b27a42683f44c8f882530d469ee4ed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b27a42683f44c8f882530d469ee4ed6" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35635 } } }
I20260620 07:19:14.925443 25447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:14.925874 25455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:19:14.927209 25446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b27a42683f44c8f882530d469ee4ed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b27a42683f44c8f882530d469ee4ed6" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35635 } } }
I20260620 07:19:14.927313 25446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6 [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:14.927448 25444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:14.910867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.11.123.190:35635)
0620 07:19:14.911274 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:14.911292 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:14.911326 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:19:14.912004 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:14.912021 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:14.912426 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:14.912801 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:14.913680 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:14.914715 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:14.915912 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:14.916510 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:14.916716 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:14.920344 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:14.920350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:14.920366 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:14.920420 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:14.922768 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:14.923302 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:14.923308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:14.923316 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:14.923463 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:14.924598 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:14.924601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:14.924603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:14.925129 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:14.926104 (+ 975us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:14.926252 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 07:19:14.926535 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
I20260620 07:19:14.929366 25455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:19:14.931075 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:19:14.935361 25455 catalog_manager.cc:1347] Generated new cluster ID: f071c4f853ab4e83bd06a7af53292169
I20260620 07:19:14.935425 25455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:19:14.948444 25455 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:19:14.949350 25455 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:19:14.954847 25455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b27a42683f44c8f882530d469ee4ed6: Generated new TSK 0
I20260620 07:19:14.955436 25455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:19:14.986858 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 25373
2026-06-20T07:19:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1400 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:19:15 dist-test-slave-j663 krb5kdc[25468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:19:15 dist-test-slave-j663 krb5kdc[25468](info): set up 2 sockets
Jun 20 07:19:15 dist-test-slave-j663 krb5kdc[25468](info): commencing operation
krb5kdc: starting...
W20260620 07:19:16.029206 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:19:16 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:19:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:19:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:19:16.144168 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:45003
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:45003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:16.250051 25481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:16.250301 25481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:16.250340 25481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:16.254191 25481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:19:16.254237 25481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:16.254256 25481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:16.254277 25481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:19:16.254292 25481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:19:16.259279 25481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:45003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:45003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:16.260169 25481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:16.261102 25481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:16.269268 25489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:16.269276 25487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:16.269271 25486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:16.269311 25481 server_base.cc:1034] running on GCE node
I20260620 07:19:16.270004 25481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:16.271078 25481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:16.272264 25481 hybrid_clock.cc:648] HybridClock initialized: now 1781939956272254 us; error 28 us; skew 500 ppm
Jun 20 07:19:16 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939956, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:16.278834 25481 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:19:16.279644 25481 webserver.cc:460] Webserver started at http://127.11.123.190:45431/ using document root <none> and password file <none>
I20260620 07:19:16.280126 25481 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:16.280184 25481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:16.280345 25481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:19:16.282379 25481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "c9deb216b2f0b63d75f2707283a61cbd"
server_key_iv: "69f12d11730d57a817f97a0916298f9c"
server_key_version: "encryptionkey@0"
I20260620 07:19:16.282853 25481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "c9deb216b2f0b63d75f2707283a61cbd"
server_key_iv: "69f12d11730d57a817f97a0916298f9c"
server_key_version: "encryptionkey@0"
I20260620 07:19:16.286222 25481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:19:16.288412 25496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:16.289616 25481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:19:16.289711 25481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "c9deb216b2f0b63d75f2707283a61cbd"
server_key_iv: "69f12d11730d57a817f97a0916298f9c"
server_key_version: "encryptionkey@0"
I20260620 07:19:16.289788 25481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:16.306237 25481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:16.308859 25481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:16.309059 25481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:16.315930 25481 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:45003
I20260620 07:19:16.316202 25548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:45003 every 8 connection(s)
I20260620 07:19:16.316882 25481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:19:16.319489 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 25481
I20260620 07:19:16.319638 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:19:16.319643 25549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:16.319947 11758 external_mini_cluster.cc:1428] Setting key e3f4983c98da9c175fd85a58a98c3697
I20260620 07:19:16.326472 25549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Bootstrap starting.
Jun 20 07:19:16 dist-test-slave-j663 krb5kdc[25468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:16.329348 25549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:16.330215 25549 log.cc:824] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:16.332546 25549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: No bootstrap required, opened a new log
I20260620 07:19:16.334833 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:16.321325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.11.123.190:45003)
0620 07:19:16.321734 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:16.321749 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:16.321783 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:19:16.322665 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:16.322684 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:16.323013 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:16.323289 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:16.324154 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:16.325187 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:16.326390 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:16.326891 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:16.327008 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:16.329787 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:16.329793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:16.329809 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:16.329840 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:16.331780 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:16.332200 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:16.332205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:16.332211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:16.332352 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:16.332610 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:16.332614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:16.332615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:16.333164 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:16.333361 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:16.333563 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 07:19:16.333866 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":128,"threads_started":1}
I20260620 07:19:16.336906 25549 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } }
I20260620 07:19:16.337247 25549 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:16.337309 25549 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd0ac8b52b64f6a9834f48bba3cd8c7, State: Initialized, Role: FOLLOWER
I20260620 07:19:16.337702 25549 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } }
I20260620 07:19:16.337852 25549 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:19:16.337915 25549 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:19:16.338006 25549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:16.338912 25549 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } }
I20260620 07:19:16.339251 25549 leader_election.cc:302] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbd0ac8b52b64f6a9834f48bba3cd8c7; no voters:
I20260620 07:19:16.339576 25549 leader_election.cc:288] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:19:16.339646 25554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:19:16.339821 25554 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 1 LEADER]: Becoming Leader. State: Replica: cbd0ac8b52b64f6a9834f48bba3cd8c7, State: Running, Role: LEADER
I20260620 07:19:16.340126 25554 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } }
I20260620 07:19:16.340818 25549 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:19:16.341874 25555 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbd0ac8b52b64f6a9834f48bba3cd8c7. Latest consensus state: current_term: 1 leader_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } } }
I20260620 07:19:16.341921 25556 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } } }
I20260620 07:19:16.342082 25556 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:16.342370 25555 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:16.342514 25563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:19:16.345438 25563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:19:16.350538 25563 catalog_manager.cc:1347] Generated new cluster ID: 925a3448786c4116920931a297e06da6
I20260620 07:19:16.350600 25563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:19:16.360354 25563 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:19:16.361313 25563 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:19:16.377581 25563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Generated new TSK 0
I20260620 07:19:16.378448 25563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:19:16.411717 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:45003
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:16.519896 25575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:16.520174 25575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:16.520248 25575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:16.524215 25575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:16.524302 25575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:16.524389 25575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:19:16.529485 25575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:16.530462 25575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:16.531440 25575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:16.540591 25580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:16.540614 25581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:16.540669 25583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:16.540737 25575 server_base.cc:1034] running on GCE node
I20260620 07:19:16.541294 25575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:16.541848 25575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:16.543005 25575 hybrid_clock.cc:648] HybridClock initialized: now 1781939956542997 us; error 30 us; skew 500 ppm
Jun 20 07:19:16 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939956, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:16.545620 25575 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:19:16.546485 25575 webserver.cc:460] Webserver started at http://127.11.123.129:42751/ using document root <none> and password file <none>
I20260620 07:19:16.547044 25575 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:16.547102 25575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:16.547335 25575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:19:16.549327 25575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "5de90df5915f4190a0f9b6c3a3cbfa72"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "43fd59a68aaf48bdd397535fad28cd72"
server_key_iv: "c71acb656c9a46bcbe1b383d30c40c78"
server_key_version: "encryptionkey@0"
I20260620 07:19:16.549880 25575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "5de90df5915f4190a0f9b6c3a3cbfa72"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "43fd59a68aaf48bdd397535fad28cd72"
server_key_iv: "c71acb656c9a46bcbe1b383d30c40c78"
server_key_version: "encryptionkey@0"
I20260620 07:19:16.553412 25575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:19:16.555714 25590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:16.556927 25575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:16.557063 25575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "5de90df5915f4190a0f9b6c3a3cbfa72"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "43fd59a68aaf48bdd397535fad28cd72"
server_key_iv: "c71acb656c9a46bcbe1b383d30c40c78"
server_key_version: "encryptionkey@0"
I20260620 07:19:16.557214 25575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:16.578104 25575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:16.581164 25575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:16.581357 25575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:16.581952 25575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:16.582734 25575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:16.582814 25575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:16.582885 25575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:16.582933 25575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:16.591537 25575 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:45231
I20260620 07:19:16.591562 25703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:45231 every 8 connection(s)
I20260620 07:19:16.592381 25575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:19:16.597581 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 25575
I20260620 07:19:16.597685 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:19:16.597981 11758 external_mini_cluster.cc:1428] Setting key 69d7738ca0856297f9bd79758702e758
Jun 20 07:19:16 dist-test-slave-j663 krb5kdc[25468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939956, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:16.606886 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:16.594165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:53299 (local address 127.11.123.190:45003)
0620 07:19:16.594303 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:16.594306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:16.595173 (+ 867us) server_negotiation.cc:407] Connection header received
0620 07:19:16.596516 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:16.596520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:16.596577 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:16.596682 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:16.597958 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:16.598623 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:16.599843 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:16.600089 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:16.600201 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:16.602896 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:16.602903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:16.602909 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:16.602943 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:16.604614 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:16.605300 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:16.605303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:16.605304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:16.605379 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:16.605793 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:16.605795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:16.605796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:16.605964 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:16.606038 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:16.606665 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 07:19:16.606769 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:19:16.608059 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:16.594576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:53299)
0620 07:19:16.595024 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:16.595065 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:16.596299 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:16.596849 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:16.596856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:16.597256 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:16.597810 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:16.597827 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:16.598734 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:16.598737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:16.599710 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:16.599719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:16.600238 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:16.600242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:16.600411 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:16.601040 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:16.601064 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:16.602729 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:16.604734 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:16.604743 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:16.604761 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:16.605197 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:16.605479 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:16.605484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:16.605489 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:16.605683 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:16.606080 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:16.606088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:16.606403 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 07:19:16.606612 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 07:19:16.606971 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:19:16.609360 25704 heartbeater.cc:347] Connected to a master server at 127.11.123.190:45003
I20260620 07:19:16.609606 25704 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:16.610080 25704 heartbeater.cc:511] Master 127.11.123.190:45003 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
I20260620 07:19:16.611764 25513 ts_manager.cc:194] Registered new tserver with Master: 5de90df5915f4190a0f9b6c3a3cbfa72 (127.11.123.129:45231)
I20260620 07:19:16.613164 25513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:53299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:19:16.627490 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:45003
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:16.734380 25709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:16.734686 25709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:16.734771 25709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:16.738786 25709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:16.738876 25709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:16.738961 25709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:19:16.744269 25709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:16.745301 25709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:16.746294 25709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:16.755671 25714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:16.755703 25715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:16.755705 25717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:16.756021 25709 server_base.cc:1034] running on GCE node
I20260620 07:19:16.756480 25709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:16.757162 25709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:16.758343 25709 hybrid_clock.cc:648] HybridClock initialized: now 1781939956758334 us; error 30 us; skew 500 ppm
Jun 20 07:19:16 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939956, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:16.761011 25709 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:19:16.761936 25709 webserver.cc:460] Webserver started at http://127.11.123.130:34255/ using document root <none> and password file <none>
I20260620 07:19:16.762444 25709 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:16.762499 25709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:16.762668 25709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:19:16.764516 25709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "4cdd35b7d35a4193991b1724c3cb492b"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "6b2cc145dad64462286acf33f0e708f8"
server_key_iv: "9028c9103552d5ab73de926be1c68967"
server_key_version: "encryptionkey@0"
I20260620 07:19:16.765012 25709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "4cdd35b7d35a4193991b1724c3cb492b"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "6b2cc145dad64462286acf33f0e708f8"
server_key_iv: "9028c9103552d5ab73de926be1c68967"
server_key_version: "encryptionkey@0"
I20260620 07:19:16.768826 25709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 07:19:16.771271 25724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:16.772498 25709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:16.772636 25709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "4cdd35b7d35a4193991b1724c3cb492b"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "6b2cc145dad64462286acf33f0e708f8"
server_key_iv: "9028c9103552d5ab73de926be1c68967"
server_key_version: "encryptionkey@0"
I20260620 07:19:16.772738 25709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:16.800365 25709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:16.803524 25709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:16.803747 25709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:16.804344 25709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:16.805225 25709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:16.805274 25709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:16.805341 25709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:16.805382 25709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:16.814057 25709 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:46869
I20260620 07:19:16.814121 25837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:46869 every 8 connection(s)
I20260620 07:19:16.814942 25709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:19:16.823462 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 25709
I20260620 07:19:16.823590 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:19:16.823899 11758 external_mini_cluster.cc:1428] Setting key 4106eb6ff0fc6e480240e519dacd22d2
Jun 20 07:19:16 dist-test-slave-j663 krb5kdc[25468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939956, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:16.828847 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:16.816727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:51071 (local address 127.11.123.190:45003)
0620 07:19:16.816884 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:16.816888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:16.817767 (+ 879us) server_negotiation.cc:407] Connection header received
0620 07:19:16.819217 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:16.819221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:16.819278 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:16.819389 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:16.820543 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:16.821123 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:16.822249 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:16.822479 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:16.822594 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:16.825290 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:16.825293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:16.825296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:16.825324 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:16.826698 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:16.827301 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:16.827308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:16.827310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:16.827409 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:16.827728 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:16.827732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:16.827735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:16.827979 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:16.828086 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:16.828548 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 07:19:16.828667 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:19:16.829736 25840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:16.817071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:51071)
0620 07:19:16.817605 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:16.817646 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:16.818996 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:16.819569 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:16.819575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:16.819930 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:16.820402 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:16.820415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:16.821262 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:16.821266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:16.822122 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:16.822131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:16.822630 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:16.822633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:16.822806 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:16.823342 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:16.823365 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:16.825131 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:16.826814 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:16.826823 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:16.826841 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:16.827203 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:16.827504 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:16.827506 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:16.827508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:16.827635 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:16.828080 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:16.828084 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:16.828299 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 07:19:16.828497 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 07:19:16.828805 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":124,"threads_started":1}
I20260620 07:19:16.830875 25838 heartbeater.cc:347] Connected to a master server at 127.11.123.190:45003
I20260620 07:19:16.831140 25838 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:16.831609 25838 heartbeater.cc:511] Master 127.11.123.190:45003 requested a full tablet report, sending...
I20260620 07:19:16.832638 25513 ts_manager.cc:194] Registered new tserver with Master: 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869)
I20260620 07:19:16.833500 25513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:51071
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:19:16.851586 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:45003
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:16.961308 25843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:16.961535 25843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:16.961622 25843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:16.965781 25843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:16.965829 25843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:16.965926 25843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:19:16.970985 25843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:16.971944 25843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:16.972884 25843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:16.982398 25849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:16.982379 25848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:16.982370 25851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:16.982506 25843 server_base.cc:1034] running on GCE node
I20260620 07:19:16.983011 25843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:16.983636 25843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:16.984822 25843 hybrid_clock.cc:648] HybridClock initialized: now 1781939956984804 us; error 30 us; skew 500 ppm
Jun 20 07:19:16 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939956, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:16.987531 25843 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:19:16.988310 25843 webserver.cc:460] Webserver started at http://127.11.123.131:33377/ using document root <none> and password file <none>
I20260620 07:19:16.988848 25843 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:16.988936 25843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:16.989216 25843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:19:16.991168 25843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "d0fe3238e0c04f20b5a31c5b086f72b1"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "7bcd7f774873b8ca7ae29394be4c137f"
server_key_iv: "89aa6e115996e77618c2490cb228a942"
server_key_version: "encryptionkey@0"
I20260620 07:19:16.991724 25843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "d0fe3238e0c04f20b5a31c5b086f72b1"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "7bcd7f774873b8ca7ae29394be4c137f"
server_key_iv: "89aa6e115996e77618c2490cb228a942"
server_key_version: "encryptionkey@0"
I20260620 07:19:16.995447 25843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 07:19:16.997722 25858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:16.998890 25843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 07:19:16.999022 25843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "d0fe3238e0c04f20b5a31c5b086f72b1"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "7bcd7f774873b8ca7ae29394be4c137f"
server_key_iv: "89aa6e115996e77618c2490cb228a942"
server_key_version: "encryptionkey@0"
I20260620 07:19:16.999126 25843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:17.024143 25843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:17.027460 25843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:17.027665 25843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:17.028245 25843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:17.029057 25843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:17.029177 25843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:17.029250 25843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:17.029291 25843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:17.037761 25843 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:42279
I20260620 07:19:17.037783 25971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:42279 every 8 connection(s)
I20260620 07:19:17.038635 25843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:19:17.047544 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 25843
I20260620 07:19:17.047662 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:19:17.047992 11758 external_mini_cluster.cc:1428] Setting key 51e7555d625992e050c8b9be94663955
Jun 20 07:19:17 dist-test-slave-j663 krb5kdc[25468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939956, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:17.052076 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.040369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:51081 (local address 127.11.123.190:45003)
0620 07:19:17.040506 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:17.040510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:17.041444 (+ 934us) server_negotiation.cc:407] Connection header received
0620 07:19:17.042835 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:17.042837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:17.042884 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:17.042956 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:17.044067 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.044579 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.045747 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.045960 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.046042 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.048697 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:17.048701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:17.048703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:17.048731 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:17.050414 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:17.050868 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:17.050870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:17.050871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:17.050941 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:17.051218 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:17.051220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:17.051221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:17.051394 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:17.051468 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:17.051874 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 07:19:17.051963 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:19:17.052915 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.040752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:51081)
0620 07:19:17.041241 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.041282 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:17.042648 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:17.043115 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:17.043121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:17.043458 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:17.043939 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.043953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.044680 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.044682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.045605 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.045614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.046087 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.046089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.046225 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.046744 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:17.046760 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:17.048514 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:17.050510 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:17.050514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:17.050524 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:17.050784 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:17.051014 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:17.051016 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:17.051018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:17.051142 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:17.051475 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:17.051480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:17.051682 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 07:19:17.051827 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 07:19:17.052051 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
I20260620 07:19:17.053937 25972 heartbeater.cc:347] Connected to a master server at 127.11.123.190:45003
I20260620 07:19:17.054157 25972 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:17.054592 25972 heartbeater.cc:511] Master 127.11.123.190:45003 requested a full tablet report, sending...
I20260620 07:19:17.055552 25513 ts_manager.cc:194] Registered new tserver with Master: d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279)
I20260620 07:19:17.056052 25513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:51081
I20260620 07:19:17.061870 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:19:17.070246 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.063633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.11.123.190:45003)
0620 07:19:17.063803 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:17.063807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:17.063936 (+ 129us) server_negotiation.cc:407] Connection header received
0620 07:19:17.064021 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:17.064023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:17.064059 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:17.064117 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:17.064718 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.065259 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.066118 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.066318 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.066416 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.067733 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:17.067736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:17.067738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:17.067753 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:17.068880 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:17.069282 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:17.069284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:17.069285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:17.069366 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:17.069564 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:17.069566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:17.069567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:17.069738 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:17.069831 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:17.069969 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 07:19:17.070050 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:19:17.072817 25513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:19:17.075685 25513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:19:17.089432 25984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.083630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:42279 (local address 127.0.0.1:45988)
0620 07:19:17.084216 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.084229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:17.084309 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:17.085061 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:17.085066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:17.085108 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:17.085233 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.085240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.086664 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.086667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.087850 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.087860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.088895 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.088899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.088998 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.089030 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:19:17.089158 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 07:19:17.089271 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":22,"thread_start_us":113,"threads_started":1}
I20260620 07:19:17.089453 25987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.084290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.11.123.131:42279)
0620 07:19:17.084602 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:17.084613 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:17.084642 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:19:17.084699 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:17.084703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:17.084887 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:17.085068 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:17.085382 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.086527 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.087983 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.088780 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.088906 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.089031 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:17.089206 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 07:19:17.089306 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":138,"threads_started":1}
I20260620 07:19:17.089771 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.083648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.11.123.129:45231)
0620 07:19:17.084988 (+ 1340us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:17.085006 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:17.085028 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:19:17.085132 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:17.085138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:17.085325 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:17.085727 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:17.085838 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.086694 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.088119 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.088770 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.088916 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.089041 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:17.089506 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 07:19:17.089624 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1235,"thread_start_us":111,"threads_started":1}
I20260620 07:19:17.089991 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.083544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45231 (local address 127.0.0.1:47742)
0620 07:19:17.083851 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.083894 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:17.084005 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:17.085539 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:17.085542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:17.085557 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:17.085633 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.085637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.086830 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.086834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.087987 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.087997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.089169 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.089172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.089268 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.089288 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:19:17.089356 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 07:19:17.089409 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":44,"thread_start_us":90,"threads_started":1}
I20260620 07:19:17.090226 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.084251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:46869 (local address 127.0.0.1:57896)
0620 07:19:17.085329 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.085343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:17.085428 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:17.085916 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:17.085919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:17.085936 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:17.086026 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.086032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.087369 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.087373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.088543 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.088555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.089781 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.089784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.089912 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.089935 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:19:17.090033 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 07:19:17.090102 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":53,"thread_start_us":95,"threads_started":1}
I20260620 07:19:17.090274 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.084430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.11.123.130:46869)
0620 07:19:17.084761 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:17.084772 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:17.085517 (+ 745us) server_negotiation.cc:407] Connection header received
0620 07:19:17.085596 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:17.085601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:17.085764 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:17.085918 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:17.086139 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.087226 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.088680 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.089503 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.089621 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.089728 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:17.090061 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 07:19:17.090146 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":159,"threads_started":1}
I20260620 07:19:17.092649 25638 tablet_service.cc:1467] Processing CreateTablet for tablet d299983bec9841f19d84f947dd94595f (DEFAULT_TABLE table=test-table [id=e2d43fda5eaf4c3fa80502bfc3a5d8b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:17.093685 25638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d299983bec9841f19d84f947dd94595f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:17.094832 25906 tablet_service.cc:1467] Processing CreateTablet for tablet d299983bec9841f19d84f947dd94595f (DEFAULT_TABLE table=test-table [id=e2d43fda5eaf4c3fa80502bfc3a5d8b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:17.095736 25906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d299983bec9841f19d84f947dd94595f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:17.099632 25772 tablet_service.cc:1467] Processing CreateTablet for tablet d299983bec9841f19d84f947dd94595f (DEFAULT_TABLE table=test-table [id=e2d43fda5eaf4c3fa80502bfc3a5d8b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:17.100217 25989 tablet_bootstrap.cc:492] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Bootstrap starting.
I20260620 07:19:17.100605 25772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d299983bec9841f19d84f947dd94595f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:17.101804 25990 tablet_bootstrap.cc:492] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1: Bootstrap starting.
I20260620 07:19:17.102636 25989 tablet_bootstrap.cc:654] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:17.104146 25989 log.cc:824] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:17.104432 25990 tablet_bootstrap.cc:654] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:17.105286 25990 log.cc:824] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:17.106673 25989 tablet_bootstrap.cc:492] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: No bootstrap required, opened a new log
I20260620 07:19:17.106865 25989 ts_tablet_manager.cc:1397] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 07:19:17.107059 25993 tablet_bootstrap.cc:492] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Bootstrap starting.
I20260620 07:19:17.107275 25990 tablet_bootstrap.cc:492] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1: No bootstrap required, opened a new log
I20260620 07:19:17.107455 25990 ts_tablet_manager.cc:1397] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 07:19:17.109309 25993 tablet_bootstrap.cc:654] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:17.109934 25993 log.cc:824] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:17.110785 25989 raft_consensus.cc:348] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.111050 25989 raft_consensus.cc:374] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:17.111106 25989 raft_consensus.cc:729] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de90df5915f4190a0f9b6c3a3cbfa72, State: Initialized, Role: FOLLOWER
I20260620 07:19:17.111174 25990 raft_consensus.cc:348] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.111398 25990 raft_consensus.cc:374] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:17.111454 25990 raft_consensus.cc:729] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fe3238e0c04f20b5a31c5b086f72b1, State: Initialized, Role: FOLLOWER
I20260620 07:19:17.111616 25993 tablet_bootstrap.cc:492] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: No bootstrap required, opened a new log
I20260620 07:19:17.111565 25989 consensus_queue.cc:260] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.111819 25993 ts_tablet_manager.cc:1397] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 07:19:17.111874 25990 consensus_queue.cc:260] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.112272 25704 heartbeater.cc:503] Master 127.11.123.190:45003 was elected leader, sending a full tablet report...
I20260620 07:19:17.112519 25989 ts_tablet_manager.cc:1428] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 07:19:17.113626 25972 heartbeater.cc:503] Master 127.11.123.190:45003 was elected leader, sending a full tablet report...
I20260620 07:19:17.113819 25990 ts_tablet_manager.cc:1428] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260620 07:19:17.115429 25993 raft_consensus.cc:348] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.115705 25993 raft_consensus.cc:374] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:17.115764 25993 raft_consensus.cc:729] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cdd35b7d35a4193991b1724c3cb492b, State: Initialized, Role: FOLLOWER
I20260620 07:19:17.116175 25993 consensus_queue.cc:260] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.116923 25993 ts_tablet_manager.cc:1428] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260620 07:19:17.116976 25838 heartbeater.cc:503] Master 127.11.123.190:45003 was elected leader, sending a full tablet report...
I20260620 07:19:17.246503 25996 raft_consensus.cc:482] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:19:17.246745 25996 raft_consensus.cc:504] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.247862 25996 leader_election.cc:288] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5de90df5915f4190a0f9b6c3a3cbfa72 (127.11.123.129:45231), 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869)
I20260620 07:19:17.251729 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.248376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:45039 (local address 127.11.123.130:46869)
0620 07:19:17.248496 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:17.248501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:17.248516 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:19:17.248572 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:17.248575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:17.248638 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:17.248722 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:17.249030 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.249658 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.250736 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.251153 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.251245 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.251326 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:17.251522 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 07:19:17.251600 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:19:17.251753 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.248271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:47345 (local address 127.11.123.129:45231)
0620 07:19:17.248408 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:17.248412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:17.248704 (+ 292us) server_negotiation.cc:407] Connection header received
0620 07:19:17.248790 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:17.248793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:17.248843 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:17.248929 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:17.249238 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.249774 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.250784 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.251180 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.251315 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.251365 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:17.251577 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 07:19:17.251645 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:19:17.251753 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.248133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:46869 (local address 127.11.123.131:45039)
0620 07:19:17.248249 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.248265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:17.248358 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:17.248753 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:17.248756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:17.248778 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:17.248910 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.248920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.249771 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.249775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.250594 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.250603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.251273 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.251278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.251382 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.251408 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 07:19:17.251478 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 07:19:17.251569 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 07:19:17.251742 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.248149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45231 (local address 127.11.123.131:47345)
0620 07:19:17.248575 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.248589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:17.248701 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:17.248948 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:17.248950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:17.248964 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:17.249125 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.249132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.249895 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.249897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.250637 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.250645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.251330 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.251334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.251429 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.251446 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 07:19:17.251542 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 07:19:17.251602 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":108,"thread_start_us":133,"threads_started":1}
I20260620 07:19:17.252475 25658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d299983bec9841f19d84f947dd94595f" candidate_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" is_pre_election: true
I20260620 07:19:17.252483 25792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d299983bec9841f19d84f947dd94595f" candidate_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cdd35b7d35a4193991b1724c3cb492b" is_pre_election: true
I20260620 07:19:17.252765 25658 raft_consensus.cc:2454] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0fe3238e0c04f20b5a31c5b086f72b1 in term 0.
I20260620 07:19:17.252784 25792 raft_consensus.cc:2454] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0fe3238e0c04f20b5a31c5b086f72b1 in term 0.
I20260620 07:19:17.253270 25862 leader_election.cc:302] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4cdd35b7d35a4193991b1724c3cb492b, 5de90df5915f4190a0f9b6c3a3cbfa72, d0fe3238e0c04f20b5a31c5b086f72b1; no voters:
I20260620 07:19:17.253511 25996 raft_consensus.cc:2781] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:19:17.253592 25996 raft_consensus.cc:482] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:19:17.253633 25996 raft_consensus.cc:3037] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:17.254594 25996 raft_consensus.cc:504] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.254917 25996 leader_election.cc:288] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [CANDIDATE]: Term 1 election: Requested vote from peers 5de90df5915f4190a0f9b6c3a3cbfa72 (127.11.123.129:45231), 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869)
I20260620 07:19:17.255206 25658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d299983bec9841f19d84f947dd94595f" candidate_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72"
I20260620 07:19:17.255244 25792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d299983bec9841f19d84f947dd94595f" candidate_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cdd35b7d35a4193991b1724c3cb492b"
I20260620 07:19:17.255343 25658 raft_consensus.cc:3037] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:17.255389 25792 raft_consensus.cc:3037] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:17.256347 25792 raft_consensus.cc:2454] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0fe3238e0c04f20b5a31c5b086f72b1 in term 1.
I20260620 07:19:17.256351 25658 raft_consensus.cc:2454] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0fe3238e0c04f20b5a31c5b086f72b1 in term 1.
I20260620 07:19:17.256613 25860 leader_election.cc:302] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cdd35b7d35a4193991b1724c3cb492b, d0fe3238e0c04f20b5a31c5b086f72b1; no voters:
I20260620 07:19:17.256820 25996 raft_consensus.cc:2781] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:19:17.256992 25996 raft_consensus.cc:686] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 1 LEADER]: Becoming Leader. State: Replica: d0fe3238e0c04f20b5a31c5b086f72b1, State: Running, Role: LEADER
I20260620 07:19:17.257318 25996 consensus_queue.cc:237] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.260530 25512 catalog_manager.cc:5495] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131). New cstate: current_term: 1 leader_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } health_report { overall_health: UNKNOWN } } }
I20260620 07:19:17.283951 25987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.280620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.11.123.131:42279)
0620 07:19:17.280724 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:17.280727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:17.280739 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:19:17.280889 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:17.280895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:17.280955 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:17.281038 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:19:17.281368 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.281870 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.282661 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.282862 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.282935 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.283165 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:19:17.283533 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:19:17.283618 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:17.283707 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 07:19:17.283753 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 07:19:17.289359 25792 raft_consensus.cc:1261] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [term 1 FOLLOWER]: Refusing update from remote peer d0fe3238e0c04f20b5a31c5b086f72b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:19:17.289407 25658 raft_consensus.cc:1261] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Refusing update from remote peer d0fe3238e0c04f20b5a31c5b086f72b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 07:19:17.289495 25973 tablet.cc:2369] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:19:17.290045 25996 consensus_queue.cc:1035] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:19:17.290201 25999 consensus_queue.cc:1035] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:19:17.297261 26003 mvcc.cc:205] Tried to move back new op lower bound from 7298826065050574848 to 7298826064928534528. Current Snapshot: MvccSnapshot[applied={T|T < 7298826065050574848}]
I20260620 07:19:17.300984 26004 mvcc.cc:205] Tried to move back new op lower bound from 7298826065050574848 to 7298826064928534528. Current Snapshot: MvccSnapshot[applied={T|T < 7298826065050574848}]
I20260620 07:19:17.305776 25512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:19:17.305953 25512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:19:17.308284 25512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Sending DeleteTablet for 3 replicas of tablet d299983bec9841f19d84f947dd94595f
I20260620 07:19:17.308972 25638 tablet_service.cc:1514] Processing DeleteTablet for tablet d299983bec9841f19d84f947dd94595f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:17 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:47742
I20260620 07:19:17.309016 25906 tablet_service.cc:1514] Processing DeleteTablet for tablet d299983bec9841f19d84f947dd94595f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:17 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:45988
I20260620 07:19:17.309213 25772 tablet_service.cc:1514] Processing DeleteTablet for tablet d299983bec9841f19d84f947dd94595f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:17 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:57896
I20260620 07:19:17.309384 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 25575
W20260620 07:19:17.315487 25500 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:45231 (error 108)
I20260620 07:19:17.316587 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 25709
I20260620 07:19:17.317462 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.316219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45231 (local address 127.0.0.1:47752)
0620 07:19:17.316429 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.316529 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 07:19:17.317780 25500 catalog_manager.cc:4584] TS 5de90df5915f4190a0f9b6c3a3cbfa72 (127.11.123.129:45231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d299983bec9841f19d84f947dd94595f: Network error: Client connection negotiation failed: client connection to 127.11.123.129:45231: connect: Connection refused (error 111)
I20260620 07:19:17.321552 26012 tablet_replica.cc:330] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1: stopping tablet replica
I20260620 07:19:17.321864 26012 raft_consensus.cc:2229] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:19:17.322293 26012 raft_consensus.cc:2258] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:19:17.323794 26012 ts_tablet_manager.cc:1905] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:19:17.327399 26012 ts_tablet_manager.cc:1918] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:19:17.327574 26012 log.cc:1196] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/d299983bec9841f19d84f947dd94595f
I20260620 07:19:17.328011 26012 ts_tablet_manager.cc:1939] T d299983bec9841f19d84f947dd94595f P d0fe3238e0c04f20b5a31c5b086f72b1: Deleting consensus metadata
I20260620 07:19:17.328531 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 25843
I20260620 07:19:17.329288 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.328678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:46869 (local address 127.0.0.1:57900)
0620 07:19:17.329157 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.329208 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.130:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388}
I20260620 07:19:17.335614 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 25481
I20260620 07:19:17.335796 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.335552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:42279 (local address 127.0.0.1:46012)
0620 07:19:17.335683 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.335730 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.131:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 07:19:17.341686 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:45003
--webserver_interface=127.11.123.190
--webserver_port=45431
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:45003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:17.449860 26015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:17.450091 26015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:17.450129 26015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:17.454018 26015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:19:17.454063 26015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:17.454082 26015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:17.454098 26015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:19:17.454147 26015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:19:17.459225 26015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:45003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:45003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=45431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:17.460170 26015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:17.461155 26015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:17.473613 26024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:17.473613 26022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:17.473642 26021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:17.473989 26015 server_base.cc:1034] running on GCE node
I20260620 07:19:17.474614 26015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:17.475723 26015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:17.476900 26015 hybrid_clock.cc:648] HybridClock initialized: now 1781939957476856 us; error 60 us; skew 500 ppm
Jun 20 07:19:17 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939957, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:17.480206 26015 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:19:17.481048 26015 webserver.cc:460] Webserver started at http://127.11.123.190:45431/ using document root <none> and password file <none>
I20260620 07:19:17.481650 26015 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:17.481753 26015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:17.485487 26015 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 07:19:17.487555 26031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:17.488701 26015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:17.488843 26015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "c9deb216b2f0b63d75f2707283a61cbd"
server_key_iv: "69f12d11730d57a817f97a0916298f9c"
server_key_version: "encryptionkey@0"
I20260620 07:19:17.489499 26015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:17.501546 26015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:17.504299 26015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:17.504505 26015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:17.511656 26015 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:45003
I20260620 07:19:17.511965 26083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:45003 every 8 connection(s)
I20260620 07:19:17.512822 26015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:19:17.517033 26084 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:19:17.517536 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 26015
I20260620 07:19:17.517967 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:45231
--local_ip_for_outbound_sockets=127.11.123.129
--tserver_master_addrs=127.11.123.190:45003
--webserver_port=42751
--webserver_interface=127.11.123.129
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:19:17.518864 26084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Bootstrap starting.
I20260620 07:19:17.535656 26084 log.cc:824] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:17.544255 26084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:19:17.544612 26084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Bootstrap complete.
I20260620 07:19:17.548621 26084 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } }
I20260620 07:19:17.549021 26084 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd0ac8b52b64f6a9834f48bba3cd8c7, State: Initialized, Role: FOLLOWER
I20260620 07:19:17.549558 26084 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } }
I20260620 07:19:17.549706 26084 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:19:17.549808 26084 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:19:17.549929 26084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:19:17.551750 26084 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } }
I20260620 07:19:17.552136 26084 leader_election.cc:302] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbd0ac8b52b64f6a9834f48bba3cd8c7; no voters:
I20260620 07:19:17.552454 26084 leader_election.cc:288] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:19:17.552532 26088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:19:17.552747 26088 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 2 LEADER]: Becoming Leader. State: Replica: cbd0ac8b52b64f6a9834f48bba3cd8c7, State: Running, Role: LEADER
I20260620 07:19:17.553022 26088 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } }
I20260620 07:19:17.553668 26084 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:19:17.554467 26090 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbd0ac8b52b64f6a9834f48bba3cd8c7. Latest consensus state: current_term: 2 leader_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } } }
I20260620 07:19:17.554497 26089 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } } }
I20260620 07:19:17.554596 26090 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:17.554610 26089 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:17.555058 26094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:19:17.559787 26094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:19:17.560511 26094 catalog_manager.cc:1259] Loaded cluster ID: 925a3448786c4116920931a297e06da6
I20260620 07:19:17.560570 26094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:19:17.562017 26094 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 07:19:17.566967 26094 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 07:19:17.572094 26094 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:19:17.572793 26094 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Loaded TSK: 0
I20260620 07:19:17.573992 26094 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:19:17.629997 26086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:17.630282 26086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:17.630360 26086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:17.634423 26086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:17.634503 26086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:17.634591 26086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:19:17.639674 26086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:45231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=42751
--tserver_master_addrs=127.11.123.190:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:17.640653 26086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:17.641685 26086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:17.650923 26112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:17.650915 26111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:17.650936 26114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:17.651393 26086 server_base.cc:1034] running on GCE node
I20260620 07:19:17.651855 26086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:17.652474 26086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:17.653647 26086 hybrid_clock.cc:648] HybridClock initialized: now 1781939957653623 us; error 38 us; skew 500 ppm
Jun 20 07:19:17 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939957, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:17.656545 26086 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:19:17.657536 26086 webserver.cc:460] Webserver started at http://127.11.123.129:42751/ using document root <none> and password file <none>
I20260620 07:19:17.658066 26086 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:17.658169 26086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:17.661957 26086 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 07:19:17.664222 26121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:17.665553 26086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:19:17.665644 26086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "5de90df5915f4190a0f9b6c3a3cbfa72"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "43fd59a68aaf48bdd397535fad28cd72"
server_key_iv: "c71acb656c9a46bcbe1b383d30c40c78"
server_key_version: "encryptionkey@0"
I20260620 07:19:17.666185 26086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:17.691708 26086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:17.694932 26086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:17.695137 26086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:17.695724 26086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:17.696856 26128 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 07:19:17.700809 26086 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:19:17.700875 26086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260620 07:19:17.700922 26086 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:19:17.702509 26086 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:19:17.702569 26086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 07:19:17.702816 26128 tablet_bootstrap.cc:492] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Bootstrap starting.
I20260620 07:19:17.713315 26086 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:45231
I20260620 07:19:17.713315 26235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:45231 every 8 connection(s)
I20260620 07:19:17.714710 26086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:19:17.723188 26128 log.cc:824] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:17.724452 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 26086
I20260620 07:19:17.724912 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:46869
--local_ip_for_outbound_sockets=127.11.123.130
--tserver_master_addrs=127.11.123.190:45003
--webserver_port=34255
--webserver_interface=127.11.123.130
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:19:17 dist-test-slave-j663 krb5kdc[25468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939957, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:17.730995 26128 tablet_bootstrap.cc:492] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:19:17.731312 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.716890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:49831)
0620 07:19:17.717409 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.717450 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:17.718233 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:17.719331 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:17.719338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:17.719744 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:17.720193 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.720206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.721555 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.721558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.722407 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.722417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.723155 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.723159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.723349 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.723953 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:17.723969 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:17.725482 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:17.728013 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:17.728019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:17.728031 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:17.728283 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:17.728708 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:17.728711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:17.728713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:17.728879 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:17.729639 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:17.729644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:17.729902 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 07:19:17.730093 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 07:19:17.730397 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":182,"threads_started":1}
I20260620 07:19:17.731400 26128 tablet_bootstrap.cc:492] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Bootstrap complete.
I20260620 07:19:17.731976 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.716838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:49831 (local address 127.11.123.190:45003)
0620 07:19:17.717233 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:17.717253 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:17.717562 (+ 309us) server_negotiation.cc:407] Connection header received
0620 07:19:17.718650 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:17.718662 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:17.718946 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:17.719278 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:17.720374 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.721398 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.722544 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.723012 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.723186 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.725688 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:17.725693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:17.725704 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:17.725732 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:17.727894 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:17.728407 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:17.728415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:17.728424 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:17.728581 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:17.728996 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:17.729002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:17.729007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:17.729507 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:17.729655 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:17.730252 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 07:19:17.730594 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":123,"threads_started":1}
I20260620 07:19:17.732666 26128 ts_tablet_manager.cc:1397] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.008s
I20260620 07:19:17.734371 26236 heartbeater.cc:347] Connected to a master server at 127.11.123.190:45003
I20260620 07:19:17.734630 26236 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:17.735212 26236 heartbeater.cc:511] Master 127.11.123.190:45003 requested a full tablet report, sending...
I20260620 07:19:17.735432 26128 raft_consensus.cc:348] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.735841 26128 raft_consensus.cc:729] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de90df5915f4190a0f9b6c3a3cbfa72, State: Initialized, Role: FOLLOWER
I20260620 07:19:17.736300 26128 consensus_queue.cc:260] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.737246 26128 ts_tablet_manager.cc:1428] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Time spent starting tablet: real 0.004s user 0.008s sys 0.000s
I20260620 07:19:17.739328 26048 ts_manager.cc:194] Registered new tserver with Master: 5de90df5915f4190a0f9b6c3a3cbfa72 (127.11.123.129:45231)
I20260620 07:19:17.742558 26048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:49831
I20260620 07:19:17.745033 26236 heartbeater.cc:503] Master 127.11.123.190:45003 was elected leader, sending a full tablet report...
Jun 20 07:19:17 dist-test-slave-j663 krb5kdc[25468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939957, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.129@KRBTEST.COM
I20260620 07:19:17.752224 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.741987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45231 (local address 127.0.0.1:47756)
0620 07:19:17.742666 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.742718 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:17.742848 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:17.743384 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:17.743390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:17.743741 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:17.743844 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.743851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.744828 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.744833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.745638 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.745648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.746464 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.746468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.746568 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.747150 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:17.747162 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:17.748641 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:17.750745 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:17.750753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:17.750756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:17.750966 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:17.751258 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:17.751260 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:17.751261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:17.751333 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:17.751778 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:17.751782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:17.751877 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 07:19:17.751983 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 07:19:17.752111 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":81,"threads_started":1}
I20260620 07:19:17.752386 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.742078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.11.123.129:45231)
0620 07:19:17.742418 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:17.742430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:17.742821 (+ 391us) server_negotiation.cc:407] Connection header received
0620 07:19:17.743005 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:17.743015 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:17.743235 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:17.743381 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:17.743971 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.744713 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.745769 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.746280 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.746394 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.748789 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:17.748798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:17.748809 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:17.748835 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:17.750644 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:17.751074 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:17.751080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:17.751082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:17.751170 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:17.751418 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:17.751421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:17.751422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:17.751675 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:17.751798 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:17.752083 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 07:19:17.752230 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":103,"threads_started":1}
I20260620 07:19:17.753327 26170 tablet_service.cc:1514] Processing DeleteTablet for tablet d299983bec9841f19d84f947dd94595f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:17 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:47756
I20260620 07:19:17.754130 26245 tablet_replica.cc:330] stopping tablet replica
I20260620 07:19:17.754485 26245 raft_consensus.cc:2229] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:19:17.754738 26245 raft_consensus.cc:2258] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:19:17.755700 26245 ts_tablet_manager.cc:1905] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:19:17.759183 26245 ts_tablet_manager.cc:1918] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:19:17.759263 26245 log.cc:1196] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal/wals/d299983bec9841f19d84f947dd94595f
I20260620 07:19:17.759534 26245 ts_tablet_manager.cc:1939] T d299983bec9841f19d84f947dd94595f P 5de90df5915f4190a0f9b6c3a3cbfa72: Deleting consensus metadata
I20260620 07:19:17.761129 26035 catalog_manager.cc:4841] TS 5de90df5915f4190a0f9b6c3a3cbfa72 (127.11.123.129:45231): tablet d299983bec9841f19d84f947dd94595f (table test-table [id=e2d43fda5eaf4c3fa80502bfc3a5d8b2]) successfully deleted
W20260620 07:19:17.839530 26241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:17.839812 26241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:17.839896 26241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:17.843832 26241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:17.843936 26241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:17.844025 26241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:19:17.849022 26241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:46869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=34255
--tserver_master_addrs=127.11.123.190:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:17.850018 26241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:17.850996 26241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:17.860325 26250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:17.860344 26253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:17.860378 26251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:17.860675 26241 server_base.cc:1034] running on GCE node
I20260620 07:19:17.861213 26241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:17.861852 26241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:17.863054 26241 hybrid_clock.cc:648] HybridClock initialized: now 1781939957863037 us; error 38 us; skew 500 ppm
Jun 20 07:19:17 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939957, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:17.865715 26241 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:19:17.866532 26241 webserver.cc:460] Webserver started at http://127.11.123.130:34255/ using document root <none> and password file <none>
I20260620 07:19:17.867026 26241 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:17.867095 26241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:17.870846 26241 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:19:17.872812 26260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:17.874014 26241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:17.874111 26241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "4cdd35b7d35a4193991b1724c3cb492b"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "6b2cc145dad64462286acf33f0e708f8"
server_key_iv: "9028c9103552d5ab73de926be1c68967"
server_key_version: "encryptionkey@0"
I20260620 07:19:17.874578 26241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:17.894335 26241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:17.896868 26241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:17.897001 26241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:17.897549 26241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:17.898509 26267 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 07:19:17.902275 26241 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:19:17.902329 26241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 07:19:17.902400 26241 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:19:17.903945 26241 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:19:17.903990 26241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 07:19:17.904244 26267 tablet_bootstrap.cc:492] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Bootstrap starting.
I20260620 07:19:17.914392 26241 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:46869
I20260620 07:19:17.914435 26374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:46869 every 8 connection(s)
I20260620 07:19:17.915279 26241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:19:17.922564 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 26241
I20260620 07:19:17.922976 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:42279
--local_ip_for_outbound_sockets=127.11.123.131
--tserver_master_addrs=127.11.123.190:45003
--webserver_port=33377
--webserver_interface=127.11.123.131
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:19:17.923553 26267 log.cc:824] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:19:17 dist-test-slave-j663 krb5kdc[25468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939957, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:17.931740 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.917218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:55127 (local address 127.11.123.190:45003)
0620 07:19:17.917385 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:17.917390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:17.918112 (+ 722us) server_negotiation.cc:407] Connection header received
0620 07:19:17.919325 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:17.919328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:17.919382 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:17.919456 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:17.921057 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.921646 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.923241 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.923612 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.923697 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.927237 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:17.927242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:17.927245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:17.927280 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:17.929163 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:17.929865 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:17.929869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:17.929871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:17.929966 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:17.930408 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:17.930411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:17.930413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:17.930650 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:17.930753 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:17.931432 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 07:19:17.931579 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:19:17.932116 26267 tablet_bootstrap.cc:492] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:19:17.932493 26267 tablet_bootstrap.cc:492] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Bootstrap complete.
I20260620 07:19:17.933362 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.917344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:55127)
0620 07:19:17.917889 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.917964 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:17.919106 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:17.919686 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:17.919701 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:17.920245 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:17.920887 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.920912 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.921794 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.921798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.923085 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.923098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.923893 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.923900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.924085 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.924903 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:17.924929 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:17.926967 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:17.929311 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:17.929320 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:17.929337 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:17.929739 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:17.930078 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:17.930083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:17.930088 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:17.930288 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:17.930775 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:17.930781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:17.931126 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 07:19:17.931362 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 07:19:17.931743 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":143,"threads_started":1}
I20260620 07:19:17.933802 26267 ts_tablet_manager.cc:1397] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.014s
I20260620 07:19:17.934597 26375 heartbeater.cc:347] Connected to a master server at 127.11.123.190:45003
I20260620 07:19:17.934839 26375 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:17.935369 26375 heartbeater.cc:511] Master 127.11.123.190:45003 requested a full tablet report, sending...
I20260620 07:19:17.936290 26267 raft_consensus.cc:348] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.936744 26267 raft_consensus.cc:729] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cdd35b7d35a4193991b1724c3cb492b, State: Initialized, Role: FOLLOWER
I20260620 07:19:17.937235 26048 ts_manager.cc:194] Registered new tserver with Master: 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869)
I20260620 07:19:17.937315 26267 consensus_queue.cc:260] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:17.938390 26267 ts_tablet_manager.cc:1428] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:19:17.938472 26048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:55127
I20260620 07:19:17.940596 26375 heartbeater.cc:503] Master 127.11.123.190:45003 was elected leader, sending a full tablet report...
Jun 20 07:19:17 dist-test-slave-j663 krb5kdc[25468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939957, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.130@KRBTEST.COM
I20260620 07:19:17.947921 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.938145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:46869 (local address 127.0.0.1:57910)
0620 07:19:17.938257 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:17.938273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:17.938360 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:17.939253 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:17.939256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:17.939459 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:17.939566 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.939572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.940508 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.940513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.941380 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:17.941390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.942069 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.942073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:17.942180 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.942681 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:17.942693 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:17.943908 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:17.946022 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:17.946027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:17.946029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:17.946384 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:17.946704 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:17.946707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:17.946709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:17.946881 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:17.947430 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:17.947434 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:17.947533 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 07:19:17.947615 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:19:17.947750 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 07:19:17.948549 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:17.938415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.11.123.130:46869)
0620 07:19:17.938823 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:17.938835 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:17.938855 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:19:17.938922 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:17.938927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:17.939123 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:17.939267 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:17.939686 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.940397 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.941498 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:17.941936 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:17.942066 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:17.944041 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:17.944050 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:17.944070 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:17.944090 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:17.945888 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:17.946511 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:17.946517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:17.946519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:17.946604 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:17.947007 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:17.947011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:17.947013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:17.947299 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:17.948117 (+ 818us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:17.948219 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 07:19:17.948394 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":315,"thread_start_us":91,"threads_started":1}
I20260620 07:19:17.949529 26309 tablet_service.cc:1514] Processing DeleteTablet for tablet d299983bec9841f19d84f947dd94595f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:17 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:57910
I20260620 07:19:17.950408 26382 tablet_replica.cc:330] stopping tablet replica
I20260620 07:19:17.950695 26382 raft_consensus.cc:2229] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:19:17.950948 26382 raft_consensus.cc:2258] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:19:17.951850 26382 ts_tablet_manager.cc:1905] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:19:17.955484 26382 ts_tablet_manager.cc:1918] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:19:17.955595 26382 log.cc:1196] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/d299983bec9841f19d84f947dd94595f
I20260620 07:19:17.955865 26382 ts_tablet_manager.cc:1939] T d299983bec9841f19d84f947dd94595f P 4cdd35b7d35a4193991b1724c3cb492b: Deleting consensus metadata
I20260620 07:19:17.957381 26033 catalog_manager.cc:4841] TS 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869): tablet d299983bec9841f19d84f947dd94595f (table test-table [id=e2d43fda5eaf4c3fa80502bfc3a5d8b2]) successfully deleted
W20260620 07:19:18.036937 26379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:18.037179 26379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:18.037256 26379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:18.041142 26379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:18.041188 26379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:18.041281 26379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:19:18.046206 26379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:42279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=33377
--tserver_master_addrs=127.11.123.190:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:18.047161 26379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:18.048243 26379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:18.057869 26387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:18.057884 26388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:18.058135 26379 server_base.cc:1034] running on GCE node
W20260620 07:19:18.058879 26390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:18.059252 26379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:18.059882 26379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:18.061064 26379 hybrid_clock.cc:648] HybridClock initialized: now 1781939958061042 us; error 30 us; skew 500 ppm
Jun 20 07:19:18 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939958, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:18.063840 26379 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:19:18.064589 26379 webserver.cc:460] Webserver started at http://127.11.123.131:33377/ using document root <none> and password file <none>
I20260620 07:19:18.065119 26379 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:18.065202 26379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:18.068688 26379 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:18.070793 26397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:18.071961 26379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 07:19:18.072054 26379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "d0fe3238e0c04f20b5a31c5b086f72b1"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "7bcd7f774873b8ca7ae29394be4c137f"
server_key_iv: "89aa6e115996e77618c2490cb228a942"
server_key_version: "encryptionkey@0"
I20260620 07:19:18.072538 26379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:18.093698 26379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:18.096319 26379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:18.096460 26379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:18.096946 26379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:18.097692 26379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:18.097738 26379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:18.097774 26379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:18.097795 26379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:18.106611 26379 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:42279
I20260620 07:19:18.106689 26510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:42279 every 8 connection(s)
I20260620 07:19:18.107581 26379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:19:18.109699 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 26379
Jun 20 07:19:18 dist-test-slave-j663 krb5kdc[25468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939958, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:18.119457 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.110307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54554 (local address 127.11.123.190:45003)
0620 07:19:18.110529 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:18.110534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:18.112035 (+ 1501us) server_negotiation.cc:407] Connection header received
0620 07:19:18.112189 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:18.112192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:18.112236 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:18.112341 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:18.113049 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.113632 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.114496 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.114738 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.114851 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:18.116281 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:18.116285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:18.116288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:18.116312 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:18.117831 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:18.118227 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:18.118230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:18.118231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:18.118330 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:18.118603 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:18.118606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:18.118608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:18.118856 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:18.119065 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:18.119168 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 07:19:18.119286 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
I20260620 07:19:18.121224 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.109333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:59599 (local address 127.11.123.190:45003)
0620 07:19:18.109515 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:18.109521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:18.110387 (+ 866us) server_negotiation.cc:407] Connection header received
0620 07:19:18.111764 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:18.111767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:18.111811 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:18.111912 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:18.113036 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.113624 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.114732 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.114946 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.115054 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:18.117467 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:18.117470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:18.117471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:18.117501 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:18.119117 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:18.119729 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:18.119734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:18.119736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:18.119826 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:18.120210 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:18.120213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:18.120215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:18.120379 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:18.120480 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:18.120952 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 07:19:18.121050 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:19:18.122007 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.109731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:59599)
0620 07:19:18.110231 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:18.110282 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:18.111574 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:18.112094 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:18.112101 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:18.112429 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:18.112892 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:18.112907 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.113753 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.113757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:18.114605 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:18.114614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.115091 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.115095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:18.115253 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:18.115684 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:18.115699 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:18.117306 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:18.119234 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:18.119242 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:18.119260 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:18.119630 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:18.119922 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:18.119926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:18.119928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:18.120105 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:18.120481 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:18.120489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:18.120720 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 07:19:18.120911 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 07:19:18.121245 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":140,"threads_started":1}
I20260620 07:19:18.122977 26511 heartbeater.cc:347] Connected to a master server at 127.11.123.190:45003
I20260620 07:19:18.123211 26511 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:18.123672 26511 heartbeater.cc:511] Master 127.11.123.190:45003 requested a full tablet report, sending...
I20260620 07:19:18.124614 26048 ts_manager.cc:194] Registered new tserver with Master: d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279)
I20260620 07:19:18.125223 26048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:59599
I20260620 07:19:18.132776 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:19:18.136490 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.133369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.11.123.190:45003)
0620 07:19:18.133516 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:18.133520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:18.133624 (+ 104us) server_negotiation.cc:407] Connection header received
0620 07:19:18.133742 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:18.133744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:18.133833 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:18.133911 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:19:18.134101 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.134574 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.135340 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.135531 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.135601 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:18.135832 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:19:18.136156 (+ 324us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:19:18.136238 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:18.136290 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 07:19:18.136339 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:19:18.137063 26048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:19:18.140264 26048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:19:18.147447 26170 tablet_service.cc:1467] Processing CreateTablet for tablet bcf6149a520f4045904c53e85f432b03 (DEFAULT_TABLE table=test-table [id=bc9688f799a44490bb8e959f1011d133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:18.147689 26309 tablet_service.cc:1467] Processing CreateTablet for tablet bcf6149a520f4045904c53e85f432b03 (DEFAULT_TABLE table=test-table [id=bc9688f799a44490bb8e959f1011d133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:18.148099 26170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf6149a520f4045904c53e85f432b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:18.148236 26309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf6149a520f4045904c53e85f432b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:18.150693 26128 tablet_bootstrap.cc:492] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: Bootstrap starting.
I20260620 07:19:18.150938 26267 tablet_bootstrap.cc:492] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b: Bootstrap starting.
I20260620 07:19:18.151532 26128 tablet_bootstrap.cc:654] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:18.151674 26267 tablet_bootstrap.cc:654] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:18.152061 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.146689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:42279 (local address 127.0.0.1:46016)
0620 07:19:18.146838 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:18.146852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:18.146924 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:18.147761 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:18.147766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:18.147789 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:18.147881 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:18.147887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.149379 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.149383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:18.150556 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:18.150568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.151670 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.151674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:18.151756 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:18.151794 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 07:19:18.151872 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 07:19:18.151947 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 07:19:18.152112 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.146892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46016 (local address 127.11.123.131:42279)
0620 07:19:18.147225 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:18.147236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:18.147283 (+ 47us) server_negotiation.cc:407] Connection header received
0620 07:19:18.147362 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:18.147368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:18.147574 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:18.147778 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:18.148004 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.149232 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.150812 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.151523 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.151683 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:18.151810 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:18.151898 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 07:19:18.151984 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":149,"threads_started":1}
I20260620 07:19:18.153138 26128 tablet_bootstrap.cc:492] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: No bootstrap required, opened a new log
I20260620 07:19:18.153228 26128 ts_tablet_manager.cc:1397] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 07:19:18.153375 26267 tablet_bootstrap.cc:492] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b: No bootstrap required, opened a new log
I20260620 07:19:18.153465 26267 ts_tablet_manager.cc:1397] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 07:19:18.153677 26128 raft_consensus.cc:348] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:18.153793 26128 raft_consensus.cc:374] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:18.153841 26128 raft_consensus.cc:729] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de90df5915f4190a0f9b6c3a3cbfa72, State: Initialized, Role: FOLLOWER
I20260620 07:19:18.153878 26267 raft_consensus.cc:348] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:18.153991 26267 raft_consensus.cc:374] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:18.153965 26128 consensus_queue.cc:260] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:18.154035 26267 raft_consensus.cc:729] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cdd35b7d35a4193991b1724c3cb492b, State: Initialized, Role: FOLLOWER
I20260620 07:19:18.154145 26267 consensus_queue.cc:260] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:18.154392 26267 ts_tablet_manager.cc:1428] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:19:18.154594 26128 ts_tablet_manager.cc:1428] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:19:18.155296 26445 tablet_service.cc:1467] Processing CreateTablet for tablet bcf6149a520f4045904c53e85f432b03 (DEFAULT_TABLE table=test-table [id=bc9688f799a44490bb8e959f1011d133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:18.156420 26445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf6149a520f4045904c53e85f432b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:18.163273 26523 tablet_bootstrap.cc:492] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1: Bootstrap starting.
I20260620 07:19:18.165568 26523 tablet_bootstrap.cc:654] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1: Neither blocks nor log segments found. Creating new log.
W20260620 07:19:18.166181 26376 tablet.cc:2369] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:19:18.166329 26523 log.cc:824] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:18.168228 26523 tablet_bootstrap.cc:492] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1: No bootstrap required, opened a new log
I20260620 07:19:18.168386 26523 ts_tablet_manager.cc:1397] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 07:19:18.171375 26523 raft_consensus.cc:348] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:18.171581 26523 raft_consensus.cc:374] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:18.171615 26523 raft_consensus.cc:729] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fe3238e0c04f20b5a31c5b086f72b1, State: Initialized, Role: FOLLOWER
I20260620 07:19:18.171958 26523 consensus_queue.cc:260] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:18.172573 26511 heartbeater.cc:503] Master 127.11.123.190:45003 was elected leader, sending a full tablet report...
I20260620 07:19:18.172899 26523 ts_tablet_manager.cc:1428] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
W20260620 07:19:18.216043 26237 tablet.cc:2369] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:19:18.289855 26242 raft_consensus.cc:482] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:19:18.290083 26242 raft_consensus.cc:504] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:18.291224 26242 leader_election.cc:288] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279), 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869)
I20260620 07:19:18.296383 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.291583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:60943 (local address 127.11.123.130:46869)
0620 07:19:18.291793 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:18.291796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:18.291967 (+ 171us) server_negotiation.cc:407] Connection header received
0620 07:19:18.292137 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:18.292140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:18.292191 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:18.292286 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:18.292544 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.293357 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.295102 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.295760 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.295863 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:18.296002 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:18.296190 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 07:19:18.296263 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 07:19:18.296381 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.291528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:41833 (local address 127.11.123.131:42279)
0620 07:19:18.291717 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:18.291722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:18.291847 (+ 125us) server_negotiation.cc:407] Connection header received
0620 07:19:18.292057 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:18.292062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:18.292108 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:18.292199 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:18.292484 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.293314 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.295102 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.295702 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.295838 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:18.295889 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:18.296186 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 07:19:18.296256 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:19:18.296444 26526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.291441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:42279 (local address 127.11.123.129:41833)
0620 07:19:18.291754 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:18.291768 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:18.291910 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:18.292210 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:18.292213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:18.292230 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:18.292356 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:18.292361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.293446 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.293450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:18.294969 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:18.294980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.295846 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.295850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:18.295977 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:18.296065 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 07:19:18.296145 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:19:18.296262 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":218,"thread_start_us":70,"threads_started":1}
I20260620 07:19:18.296451 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.291450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:46869 (local address 127.11.123.129:60943)
0620 07:19:18.291878 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:18.291889 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:18.292011 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:18.292297 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:18.292300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:18.292316 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:18.292414 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:18.292420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.293452 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.293454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:18.294975 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:18.294981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.295862 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.295865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:18.296002 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:18.296077 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 07:19:18.296145 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 07:19:18.296262 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":172,"thread_start_us":92,"threads_started":1}
I20260620 07:19:18.297012 26329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf6149a520f4045904c53e85f432b03" candidate_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cdd35b7d35a4193991b1724c3cb492b" is_pre_election: true
I20260620 07:19:18.297057 26465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf6149a520f4045904c53e85f432b03" candidate_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" is_pre_election: true
I20260620 07:19:18.297361 26329 raft_consensus.cc:2454] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de90df5915f4190a0f9b6c3a3cbfa72 in term 0.
I20260620 07:19:18.297376 26465 raft_consensus.cc:2454] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de90df5915f4190a0f9b6c3a3cbfa72 in term 0.
I20260620 07:19:18.297828 26123 leader_election.cc:302] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4cdd35b7d35a4193991b1724c3cb492b, 5de90df5915f4190a0f9b6c3a3cbfa72, d0fe3238e0c04f20b5a31c5b086f72b1; no voters:
I20260620 07:19:18.298076 26242 raft_consensus.cc:2781] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:19:18.298156 26242 raft_consensus.cc:482] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:19:18.298205 26242 raft_consensus.cc:3037] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:18.299299 26242 raft_consensus.cc:504] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:18.299682 26242 leader_election.cc:288] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 1 election: Requested vote from peers d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279), 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869)
I20260620 07:19:18.299942 26465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf6149a520f4045904c53e85f432b03" candidate_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1"
I20260620 07:19:18.300045 26465 raft_consensus.cc:3037] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:18.300053 26329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf6149a520f4045904c53e85f432b03" candidate_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cdd35b7d35a4193991b1724c3cb492b"
I20260620 07:19:18.300165 26329 raft_consensus.cc:3037] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:18.301330 26465 raft_consensus.cc:2454] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de90df5915f4190a0f9b6c3a3cbfa72 in term 1.
I20260620 07:19:18.301330 26329 raft_consensus.cc:2454] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de90df5915f4190a0f9b6c3a3cbfa72 in term 1.
I20260620 07:19:18.301594 26123 leader_election.cc:302] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cdd35b7d35a4193991b1724c3cb492b, 5de90df5915f4190a0f9b6c3a3cbfa72; no voters:
I20260620 07:19:18.301782 26242 raft_consensus.cc:2781] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:19:18.302014 26242 raft_consensus.cc:686] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 LEADER]: Becoming Leader. State: Replica: 5de90df5915f4190a0f9b6c3a3cbfa72, State: Running, Role: LEADER
I20260620 07:19:18.302273 26242 consensus_queue.cc:237] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:18.304832 26048 catalog_manager.cc:5495] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5de90df5915f4190a0f9b6c3a3cbfa72 (127.11.123.129). New cstate: current_term: 1 leader_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } health_report { overall_health: UNKNOWN } } }
I20260620 07:19:18.345049 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.341436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.11.123.129:45231)
0620 07:19:18.341726 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:18.341733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:18.341744 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:19:18.341821 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:18.341823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:18.341875 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:18.342003 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:19:18.342247 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.342754 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.343604 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:18.343834 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:18.343944 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:18.344240 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:19:18.344647 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:19:18.344783 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:18.344834 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 07:19:18.344890 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
I20260620 07:19:18.349124 26465 raft_consensus.cc:1261] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1 [term 1 FOLLOWER]: Refusing update from remote peer 5de90df5915f4190a0f9b6c3a3cbfa72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:19:18.349121 26329 raft_consensus.cc:1261] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b [term 1 FOLLOWER]: Refusing update from remote peer 5de90df5915f4190a0f9b6c3a3cbfa72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:19:18.349906 26528 consensus_queue.cc:1035] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:19:18.350107 26242 consensus_queue.cc:1035] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:19:18.357584 26534 mvcc.cc:205] Tried to move back new op lower bound from 7298826069390938112 to 7298826069208653824. Current Snapshot: MvccSnapshot[applied={T|T < 7298826069390938112}]
I20260620 07:19:18.364302 26048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:19:18.364475 26048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:19:18.366844 26048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Sending DeleteTablet for 3 replicas of tablet bcf6149a520f4045904c53e85f432b03
I20260620 07:19:18.367532 26170 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf6149a520f4045904c53e85f432b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:18 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:47756
I20260620 07:19:18.367514 26445 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf6149a520f4045904c53e85f432b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:18 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:46016
I20260620 07:19:18.367867 26309 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf6149a520f4045904c53e85f432b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:18 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:57910
I20260620 07:19:18.368067 26382 tablet_replica.cc:330] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b: stopping tablet replica
I20260620 07:19:18.368065 26542 tablet_replica.cc:330] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: stopping tablet replica
I20260620 07:19:18.368093 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 26086
I20260620 07:19:18.368196 26382 raft_consensus.cc:2229] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:19:18.368341 26382 raft_consensus.cc:2258] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:19:18.370245 26543 tablet_replica.cc:330] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1: stopping tablet replica
I20260620 07:19:18.370268 26382 ts_tablet_manager.cc:1905] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:19:18.370584 26543 raft_consensus.cc:2229] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:19:18.370898 26543 raft_consensus.cc:2258] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:19:18.372327 26543 ts_tablet_manager.cc:1905] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:19:18.372813 26382 ts_tablet_manager.cc:1918] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:19:18.372895 26382 log.cc:1196] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/bcf6149a520f4045904c53e85f432b03
I20260620 07:19:18.373179 26382 ts_tablet_manager.cc:1939] T bcf6149a520f4045904c53e85f432b03 P 4cdd35b7d35a4193991b1724c3cb492b: Deleting consensus metadata
I20260620 07:19:18.373879 26033 catalog_manager.cc:4841] TS 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869): tablet bcf6149a520f4045904c53e85f432b03 (table test-table [id=bc9688f799a44490bb8e959f1011d133]) successfully deleted
I20260620 07:19:18.374807 26543 ts_tablet_manager.cc:1918] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:19:18.374927 26543 log.cc:1196] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/bcf6149a520f4045904c53e85f432b03
I20260620 07:19:18.375313 26543 ts_tablet_manager.cc:1939] T bcf6149a520f4045904c53e85f432b03 P d0fe3238e0c04f20b5a31c5b086f72b1: Deleting consensus metadata
I20260620 07:19:18.375515 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 26241
W20260620 07:19:18.375488 26035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:45231 (error 108)
I20260620 07:19:18.376659 26032 catalog_manager.cc:4841] TS d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279): tablet bcf6149a520f4045904c53e85f432b03 (table test-table [id=bc9688f799a44490bb8e959f1011d133]) successfully deleted
I20260620 07:19:18.376665 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.376309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:45231 (local address 127.0.0.1:47780)
0620 07:19:18.376494 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:18.376568 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 07:19:18.376982 26035 catalog_manager.cc:4584] TS 5de90df5915f4190a0f9b6c3a3cbfa72 (127.11.123.129:45231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcf6149a520f4045904c53e85f432b03: Network error: Client connection negotiation failed: client connection to 127.11.123.129:45231: connect: Connection refused (error 111)
I20260620 07:19:18.381819 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 26379
I20260620 07:19:18.388110 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 26015
I20260620 07:19:18.394474 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:45003
--webserver_interface=127.11.123.190
--webserver_port=45431
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:45003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:18.503218 26545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:18.503469 26545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:18.503507 26545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:18.507472 26545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:19:18.507531 26545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:18.507550 26545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:18.507567 26545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:19:18.507582 26545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:19:18.512634 26545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:45003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:45003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=45431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:18.513615 26545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:18.514577 26545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:18.522894 26550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:18.522960 26553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:18.523085 26545 server_base.cc:1034] running on GCE node
W20260620 07:19:18.522936 26551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:18.523741 26545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:18.524925 26545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:18.526100 26545 hybrid_clock.cc:648] HybridClock initialized: now 1781939958526071 us; error 45 us; skew 500 ppm
Jun 20 07:19:18 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939958, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:18.529537 26545 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:19:18.530503 26545 webserver.cc:460] Webserver started at http://127.11.123.190:45431/ using document root <none> and password file <none>
I20260620 07:19:18.531080 26545 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:18.531169 26545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:18.534962 26545 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 07:19:18.537065 26560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:18.538445 26545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:18.538595 26545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "c9deb216b2f0b63d75f2707283a61cbd"
server_key_iv: "69f12d11730d57a817f97a0916298f9c"
server_key_version: "encryptionkey@0"
I20260620 07:19:18.539157 26545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:18.558315 26545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:18.561424 26545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:18.561645 26545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:18.569002 26545 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:45003
I20260620 07:19:18.569300 26612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:45003 every 8 connection(s)
I20260620 07:19:18.570163 26545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:19:18.570624 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 26545
I20260620 07:19:18.571072 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:45231
--local_ip_for_outbound_sockets=127.11.123.129
--tserver_master_addrs=127.11.123.190:45003
--webserver_port=42751
--webserver_interface=127.11.123.129
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:19:18.574582 26613 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:19:18.576231 26613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Bootstrap starting.
I20260620 07:19:18.593693 26613 log.cc:824] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:18.605047 26613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:19:18.605576 26613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7: Bootstrap complete.
I20260620 07:19:18.610008 26613 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } }
I20260620 07:19:18.610411 26613 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd0ac8b52b64f6a9834f48bba3cd8c7, State: Initialized, Role: FOLLOWER
I20260620 07:19:18.610982 26613 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } }
I20260620 07:19:18.611107 26613 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:19:18.611198 26613 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:19:18.611323 26613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 2 FOLLOWER]: Advancing to term 3
I20260620 07:19:18.613425 26613 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } }
I20260620 07:19:18.613775 26613 leader_election.cc:302] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbd0ac8b52b64f6a9834f48bba3cd8c7; no voters:
I20260620 07:19:18.614107 26613 leader_election.cc:288] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 07:19:18.614266 26617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 07:19:18.614513 26617 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [term 3 LEADER]: Becoming Leader. State: Replica: cbd0ac8b52b64f6a9834f48bba3cd8c7, State: Running, Role: LEADER
I20260620 07:19:18.614825 26617 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } }
I20260620 07:19:18.615401 26613 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:19:18.616289 26618 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } } }
I20260620 07:19:18.616412 26618 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:18.616601 26619 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbd0ac8b52b64f6a9834f48bba3cd8c7. Latest consensus state: current_term: 3 leader_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd0ac8b52b64f6a9834f48bba3cd8c7" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 45003 } } }
I20260620 07:19:18.616683 26619 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd0ac8b52b64f6a9834f48bba3cd8c7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:18.616948 26626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:19:18.622031 26626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:19:18.622783 26626 catalog_manager.cc:1259] Loaded cluster ID: 925a3448786c4116920931a297e06da6
I20260620 07:19:18.622853 26626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 07:19:18.624059 26626 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f21340449ad google::LogMessage::Fail() at ??:0
@ 0x7f213404788a google::LogMessage::SendToLog() at ??:0
@ 0x7f21340444de google::LogMessage::Flush() at ??:0
@ 0x7f2134045a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f213e983fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f213e984d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f213e981579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f213e9c5640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f21362963d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2136296c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f21362983f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2136287512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f21378cb6db start_thread at ??:0
@ 0x7f2132b4261f clone at ??:0
W20260620 07:19:18.719944 26615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:18.720369 26615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:18.720444 26615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:18.726742 26615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:18.726840 26615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:18.726933 26615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:19:18.735265 26615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:45231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=42751
--tserver_master_addrs=127.11.123.190:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:18.736703 26615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:18.738095 26615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:18.749557 26654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:18.749751 26657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:18.749596 26655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:18.749863 26615 server_base.cc:1034] running on GCE node
I20260620 07:19:18.750427 26615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:18.751256 26615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:18.752466 26615 hybrid_clock.cc:648] HybridClock initialized: now 1781939958752425 us; error 47 us; skew 500 ppm
Jun 20 07:19:18 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939958, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:18.755873 26615 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:19:18.756726 26615 webserver.cc:460] Webserver started at http://127.11.123.129:42751/ using document root <none> and password file <none>
I20260620 07:19:18.757370 26615 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:18.757436 26615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:18.761519 26615 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:19:18.763908 26664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:18.765336 26615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:19:18.765473 26615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "5de90df5915f4190a0f9b6c3a3cbfa72"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "43fd59a68aaf48bdd397535fad28cd72"
server_key_iv: "c71acb656c9a46bcbe1b383d30c40c78"
server_key_version: "encryptionkey@0"
I20260620 07:19:18.766033 26615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:18.798671 26615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:18.802143 26615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:18.802361 26615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:18.802973 26615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:18.804109 26671 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 07:19:18.809495 26615 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:19:18.809577 26615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 07:19:18.809666 26615 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:19:18.811581 26615 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:19:18.811676 26615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 07:19:18.811980 26671 tablet_bootstrap.cc:492] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: Bootstrap starting.
I20260620 07:19:18.822877 26615 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:45231
I20260620 07:19:18.823112 26778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:45231 every 8 connection(s)
I20260620 07:19:18.823927 26615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:19:18.828670 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 26615
I20260620 07:19:18.829177 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:46869
--local_ip_for_outbound_sockets=127.11.123.130
--tserver_master_addrs=127.11.123.190:45003
--webserver_port=34255
--webserver_interface=127.11.123.130
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:19:18.835304 26671 log.cc:824] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:18.842466 26671 tablet_bootstrap.cc:492] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:19:18.842865 26671 tablet_bootstrap.cc:492] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: Bootstrap complete.
I20260620 07:19:18.844004 26671 ts_tablet_manager.cc:1397] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.003s
I20260620 07:19:18.846681 26671 raft_consensus.cc:348] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:18.847114 26671 raft_consensus.cc:729] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de90df5915f4190a0f9b6c3a3cbfa72, State: Initialized, Role: FOLLOWER
I20260620 07:19:18.847616 26671 consensus_queue.cc:260] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:18.848583 26671 ts_tablet_manager.cc:1428] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260620 07:19:18.856930 26781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:19:18.826104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:42713)
0620 07:19:18.826590 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:18.826644 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:18.827479 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:18.856158 (+ 28679us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":151,"threads_started":1}
W20260620 07:19:18.857401 26779 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:45003 (0 consecutive failures): Network error: Failed to ping master at 127.11.123.190:45003: Client connection negotiation failed: client connection to 127.11.123.190:45003: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 07:19:18.858130 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.857896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:45439)
0620 07:19:18.858021 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:18.858080 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 07:19:18.858830 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:18.858623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:56241)
0620 07:19:18.858746 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:18.858786 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 07:19:18.859057 26779 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:19:18.962400 26782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:18.962731 26782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:18.962809 26782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:18.966986 26782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:18.967046 26782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:18.967135 26782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:19:18.972637 26782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:46869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=34255
--tserver_master_addrs=127.11.123.190:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:18.973755 26782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:18.974779 26782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:18.984071 26792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:18.984071 26790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:18.984222 26789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:18.984491 26782 server_base.cc:1034] running on GCE node
I20260620 07:19:18.984881 26782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:18.985571 26782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:18.986755 26782 hybrid_clock.cc:648] HybridClock initialized: now 1781939958986737 us; error 33 us; skew 500 ppm
Jun 20 07:19:18 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939958, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:18.989475 26782 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:19:18.990361 26782 webserver.cc:460] Webserver started at http://127.11.123.130:34255/ using document root <none> and password file <none>
I20260620 07:19:18.990933 26782 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:18.991031 26782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:18.994817 26782 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 07:19:18.996927 26799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:18.998175 26782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:18.998309 26782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "4cdd35b7d35a4193991b1724c3cb492b"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "6b2cc145dad64462286acf33f0e708f8"
server_key_iv: "9028c9103552d5ab73de926be1c68967"
server_key_version: "encryptionkey@0"
I20260620 07:19:18.998883 26782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:19.028357 26782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:19.031391 26782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:19.031600 26782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:19.032205 26782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:19.033066 26782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:19.033186 26782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:19.033254 26782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:19.033304 26782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:19.042269 26782 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:46869
I20260620 07:19:19.042295 26912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:46869 every 8 connection(s)
I20260620 07:19:19.043246 26782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:19:19.047214 26915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:19.045509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:49841)
0620 07:19:19.045956 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:19.046069 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":121,"threads_started":1}
W20260620 07:19:19.047747 26913 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:45003 (0 consecutive failures): Network error: Failed to ping master at 127.11.123.190:45003: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
I20260620 07:19:19.048312 26915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:19.048104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:42021)
0620 07:19:19.048213 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:19.048260 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 07:19:19.048969 26915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:19.048782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:35977)
0620 07:19:19.048884 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:19.048923 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 07:19:19.049192 26913 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:19:19.051491 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 26782
I20260620 07:19:19.051882 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:42279
--local_ip_for_outbound_sockets=127.11.123.131
--tserver_master_addrs=127.11.123.190:45003
--webserver_port=33377
--webserver_interface=127.11.123.131
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:19.161391 26916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:19.161649 26916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:19.161692 26916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:19.165809 26916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:19.165866 26916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:19.165952 26916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:19:19.171344 26916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:42279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=33377
--tserver_master_addrs=127.11.123.190:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:19.172380 26916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:19.173410 26916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:19.182617 26924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:19.182744 26922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:19.182936 26921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:19.183137 26916 server_base.cc:1034] running on GCE node
I20260620 07:19:19.183537 26916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:19.184155 26916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:19.185338 26916 hybrid_clock.cc:648] HybridClock initialized: now 1781939959185298 us; error 45 us; skew 500 ppm
Jun 20 07:19:19 dist-test-slave-j663 krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939959, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:19.187879 26916 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:19:19.188689 26916 webserver.cc:460] Webserver started at http://127.11.123.131:33377/ using document root <none> and password file <none>
I20260620 07:19:19.189297 26916 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:19.189395 26916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:19.193043 26916 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:19:19.195127 26931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:19.196318 26916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:19.196461 26916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "d0fe3238e0c04f20b5a31c5b086f72b1"
format_stamp: "Formatted at 2026-06-20 07:19:16 on dist-test-slave-j663"
server_key: "7bcd7f774873b8ca7ae29394be4c137f"
server_key_iv: "89aa6e115996e77618c2490cb228a942"
server_key_version: "encryptionkey@0"
I20260620 07:19:19.196992 26916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:19.214404 26916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:19.217216 26916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:19.217420 26916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:19.218044 26916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:19.218855 26916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:19.218927 26916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:19.218998 26916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:19.219044 26916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:19.228020 26916 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:42279
I20260620 07:19:19.228048 27044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:42279 every 8 connection(s)
I20260620 07:19:19.228982 26916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:19:19.232679 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:19.230996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:46719)
0620 07:19:19.231444 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:19.231555 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
W20260620 07:19:19.233166 27045 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:45003 (0 consecutive failures): Network error: Failed to ping master at 127.11.123.190:45003: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
I20260620 07:19:19.233726 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:19.233508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:40643)
0620 07:19:19.233615 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:19.233663 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 07:19:19.234274 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:19.234071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:58245)
0620 07:19:19.234171 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:19.234214 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 07:19:19.234458 27045 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:19:19.238158 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 26916
I20260620 07:19:19.860152 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:19.859692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:42209)
0620 07:19:19.859981 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:19.860043 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":74,"threads_started":1}
I20260620 07:19:20.050058 27050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:20.049671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:45393)
0620 07:19:20.049911 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:20.049962 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260620 07:19:20.235404 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:20.234996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:51753)
0620 07:19:20.235235 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:20.235298 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
I20260620 07:19:20.417217 27052 raft_consensus.cc:482] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:19:20.417435 27052 raft_consensus.cc:504] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:20.418576 27052 leader_election.cc:288] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279), 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869)
Jun 20 07:19:20 dist-test-slave-j663 krb5kdc[25468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939958, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.130@KRBTEST.COM
Jun 20 07:19:20 dist-test-slave-j663 krb5kdc[25468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939958, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.131@KRBTEST.COM
I20260620 07:19:20.432900 27053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:20.418852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:46869 (local address 127.11.123.129:35073)
0620 07:19:20.419172 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:20.419183 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:20.419257 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:20.420912 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:20.420918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:20.421565 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:20.422037 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:20.422051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:20.423336 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:20.423339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:20.424248 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:20.424257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:20.425027 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:20.425031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:20.425227 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:20.425995 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:20.426007 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:20.427845 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:20.430456 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:20.430461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:20.430472 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:20.430774 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:20.431185 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:20.431188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:20.431190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:20.431304 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:20.431921 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:20.431928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:20.432167 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 07:19:20.432359 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 07:19:20.432708 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":208,"spinlock_wait_cycles":6144,"thread_start_us":69,"threads_started":1}
I20260620 07:19:20.432914 27055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:20.419013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:35073 (local address 127.11.123.130:46869)
0620 07:19:20.419444 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:20.419468 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:20.419513 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:19:20.420210 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:20.420215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:20.420550 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:20.420870 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:20.422223 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:20.423183 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:20.424368 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:20.424906 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:20.425073 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:20.428109 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:20.428116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:20.428133 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:20.428173 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:20.430316 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:20.430938 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:20.430944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:20.430949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:20.431068 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:20.431413 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:20.431416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:20.431418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:20.431802 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:20.431947 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:20.432464 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 07:19:20.432710 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":94,"threads_started":1}
I20260620 07:19:20.432930 27056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:20.419244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:34547 (local address 127.11.123.131:42279)
0620 07:19:20.419553 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:20.419571 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:20.419601 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:19:20.420209 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:20.420214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:20.420550 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:20.421031 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:20.422210 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:20.423166 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:20.424370 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:20.424920 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:20.425105 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:20.428524 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:20.428538 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:20.428550 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:20.428577 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:20.430475 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:20.430924 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:20.430930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:20.430935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:20.431059 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:20.431401 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:20.431407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:20.431408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:20.431802 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:20.431947 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:20.432465 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 07:19:20.432747 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":127,"threads_started":1}
I20260620 07:19:20.432963 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:20.418833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:42279 (local address 127.11.123.129:34547)
0620 07:19:20.419083 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:20.419098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:20.419214 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:20.420972 (+ 1758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:20.420975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:20.421566 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:20.422037 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:20.422052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:20.423329 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:20.423337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:20.424248 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:20.424257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:20.425040 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:20.425045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:20.425230 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:20.425935 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:20.425952 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:20.428322 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:20.430587 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:20.430592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:20.430594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:20.430806 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:20.431158 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:20.431162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:20.431164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:20.431300 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:20.431921 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:20.431928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:20.432166 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 07:19:20.432358 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 07:19:20.432798 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":243,"spinlock_wait_cycles":46848,"thread_start_us":84,"threads_started":1}
W20260620 07:19:20.436209 26666 leader_election.cc:341] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869): Not found: Tablet not found: bcf6149a520f4045904c53e85f432b03
W20260620 07:19:20.436476 26665 leader_election.cc:341] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279): Not found: Tablet not found: bcf6149a520f4045904c53e85f432b03
I20260620 07:19:20.436563 26665 leader_election.cc:302] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5de90df5915f4190a0f9b6c3a3cbfa72; no voters: 4cdd35b7d35a4193991b1724c3cb492b, d0fe3238e0c04f20b5a31c5b086f72b1
I20260620 07:19:20.436889 27052 raft_consensus.cc:2726] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:19:20.861429 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:20.861031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:43243)
0620 07:19:20.861268 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:20.861341 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 07:19:21.051170 27057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:21.050786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:39231)
0620 07:19:21.051024 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:21.051077 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260620 07:19:21.236549 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:21.236151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:54123)
0620 07:19:21.236411 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:21.236465 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260620 07:19:21.862576 27059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:21.862194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:33645)
0620 07:19:21.862431 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:21.862485 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260620 07:19:21.963836 27060 raft_consensus.cc:482] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:19:21.963989 27060 raft_consensus.cc:504] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:21.964457 27060 leader_election.cc:288] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279), 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869)
W20260620 07:19:21.965184 26666 leader_election.cc:341] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869): Not found: Tablet not found: bcf6149a520f4045904c53e85f432b03
W20260620 07:19:21.965389 26665 leader_election.cc:341] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279): Not found: Tablet not found: bcf6149a520f4045904c53e85f432b03
I20260620 07:19:21.965457 26665 leader_election.cc:302] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5de90df5915f4190a0f9b6c3a3cbfa72; no voters: 4cdd35b7d35a4193991b1724c3cb492b, d0fe3238e0c04f20b5a31c5b086f72b1
I20260620 07:19:21.965580 27060 raft_consensus.cc:2726] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:19:22.052309 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:22.051912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:58159)
0620 07:19:22.052164 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:22.052218 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260620 07:19:22.237674 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:22.237290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:39405)
0620 07:19:22.237531 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:22.237583 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260620 07:19:22.863710 27063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:22.863320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:47613)
0620 07:19:22.863568 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:22.863621 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
I20260620 07:19:23.053401 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:23.052983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:38347)
0620 07:19:23.053265 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:23.053315 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":130,"threads_started":1}
I20260620 07:19:23.238674 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:23.238319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:38539)
0620 07:19:23.238546 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:23.238595 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
I20260620 07:19:23.865015 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:23.864542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:56939)
0620 07:19:23.864864 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:23.864918 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":76,"threads_started":1}
I20260620 07:19:24.054453 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:24.054090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:41549)
0620 07:19:24.054318 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:24.054369 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
I20260620 07:19:24.057274 27068 raft_consensus.cc:482] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:19:24.057400 27068 raft_consensus.cc:504] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:24.057811 27068 leader_election.cc:288] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279), 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869)
W20260620 07:19:24.058485 26666 leader_election.cc:341] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869): Not found: Tablet not found: bcf6149a520f4045904c53e85f432b03
W20260620 07:19:24.058596 26665 leader_election.cc:341] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279): Not found: Tablet not found: bcf6149a520f4045904c53e85f432b03
I20260620 07:19:24.058725 26665 leader_election.cc:302] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5de90df5915f4190a0f9b6c3a3cbfa72; no voters: 4cdd35b7d35a4193991b1724c3cb492b, d0fe3238e0c04f20b5a31c5b086f72b1
I20260620 07:19:24.058878 27068 raft_consensus.cc:2726] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:19:24.239822 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:24.239425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:42515)
0620 07:19:24.239674 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:24.239732 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
I20260620 07:19:24.866230 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:24.865812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:36365)
0620 07:19:24.866081 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:24.866137 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
I20260620 07:19:25.055577 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:25.055183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:47687)
0620 07:19:25.055419 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:25.055474 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
I20260620 07:19:25.240933 27072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:25.240531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:47385)
0620 07:19:25.240786 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:25.240842 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260620 07:19:25.867323 27073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:25.866947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:41347)
0620 07:19:25.867173 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:25.867225 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260620 07:19:26.056666 27074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:26.056268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:41079)
0620 07:19:26.056523 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:26.056576 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":68,"threads_started":1}
I20260620 07:19:26.242100 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:26.241665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:46249)
0620 07:19:26.241957 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:26.242009 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":83,"threads_started":1}
I20260620 07:19:26.868535 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:26.868131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:59433)
0620 07:19:26.868383 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:26.868440 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260620 07:19:26.962548 27077 raft_consensus.cc:482] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:19:26.962687 27077 raft_consensus.cc:504] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:26.963104 27077 leader_election.cc:288] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279), 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869)
W20260620 07:19:26.963866 26665 leader_election.cc:341] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279): Not found: Tablet not found: bcf6149a520f4045904c53e85f432b03
W20260620 07:19:26.963968 26666 leader_election.cc:341] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869): Not found: Tablet not found: bcf6149a520f4045904c53e85f432b03
I20260620 07:19:26.964075 26666 leader_election.cc:302] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5de90df5915f4190a0f9b6c3a3cbfa72; no voters: 4cdd35b7d35a4193991b1724c3cb492b, d0fe3238e0c04f20b5a31c5b086f72b1
I20260620 07:19:26.964241 27077 raft_consensus.cc:2726] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:19:27.057789 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:27.057400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:52133)
0620 07:19:27.057624 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:27.057690 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
I20260620 07:19:27.243458 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:27.242937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:45773)
0620 07:19:27.243272 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:27.243332 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":88,"threads_started":1}
I20260620 07:19:27.869879 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:27.869488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:48415)
0620 07:19:27.869737 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:27.869785 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
I20260620 07:19:28.058830 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:28.058460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:55095)
0620 07:19:28.058681 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:28.058731 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260620 07:19:28.244527 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:28.244133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:56993)
0620 07:19:28.244378 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:28.244429 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
I20260620 07:19:28.871148 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:28.870709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:51921)
0620 07:19:28.871000 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:28.871055 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
I20260620 07:19:29.059924 27084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:29.059556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:49851)
0620 07:19:29.059786 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:29.059836 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
I20260620 07:19:29.245690 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:29.245321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:41717)
0620 07:19:29.245550 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:29.245602 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260620 07:19:29.872432 27086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:29.871879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:60593)
0620 07:19:29.872124 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:29.872312 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260620 07:19:30.060948 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:30.060588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:50129)
0620 07:19:30.060809 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:30.060859 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
I20260620 07:19:30.246840 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:30.246435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:35497)
0620 07:19:30.246688 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:30.246745 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260620 07:19:30.652669 27089 raft_consensus.cc:482] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:19:30.652814 27089 raft_consensus.cc:504] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fe3238e0c04f20b5a31c5b086f72b1" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 42279 } } peers { permanent_uuid: "5de90df5915f4190a0f9b6c3a3cbfa72" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 45231 } } peers { permanent_uuid: "4cdd35b7d35a4193991b1724c3cb492b" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 46869 } }
I20260620 07:19:30.653409 27089 leader_election.cc:288] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279), 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869)
W20260620 07:19:30.654206 26665 leader_election.cc:341] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0fe3238e0c04f20b5a31c5b086f72b1 (127.11.123.131:42279): Not found: Tablet not found: bcf6149a520f4045904c53e85f432b03
W20260620 07:19:30.654296 26666 leader_election.cc:341] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4cdd35b7d35a4193991b1724c3cb492b (127.11.123.130:46869): Not found: Tablet not found: bcf6149a520f4045904c53e85f432b03
I20260620 07:19:30.654410 26666 leader_election.cc:302] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5de90df5915f4190a0f9b6c3a3cbfa72; no voters: 4cdd35b7d35a4193991b1724c3cb492b, d0fe3238e0c04f20b5a31c5b086f72b1
I20260620 07:19:30.654628 27089 raft_consensus.cc:2726] T bcf6149a520f4045904c53e85f432b03 P 5de90df5915f4190a0f9b6c3a3cbfa72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:19:30.873739 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:30.873333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:39519)
0620 07:19:30.873586 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:30.873647 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260620 07:19:31.062263 27091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:31.061827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:50999)
0620 07:19:31.062104 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:31.062174 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
I20260620 07:19:31.248108 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:31.247651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:48407)
0620 07:19:31.247941 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:31.248017 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
I20260620 07:19:31.874951 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:31.874507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:54405)
0620 07:19:31.874756 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:31.874811 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260620 07:19:32.063441 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:32.062991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:44595)
0620 07:19:32.063248 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:32.063335 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260620 07:19:32.249383 27095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:32.248911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:33025)
0620 07:19:32.249235 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:32.249291 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":132,"threads_started":1}
I20260620 07:19:32.876147 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:32.875744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:37499)
0620 07:19:32.876003 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:32.876054 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260620 07:19:33.064509 27097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:33.064134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:37863)
0620 07:19:33.064367 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:33.064419 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20260620 07:19:33.250416 27098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:33.250032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:41499)
0620 07:19:33.250274 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:33.250323 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
I20260620 07:19:33.877586 27099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:33.877204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.129:33671)
0620 07:19:33.877442 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:33.877496 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260620 07:19:34.065688 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:34.065294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.130:43507)
0620 07:19:34.065546 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:34.065599 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":65,"threads_started":1}
I20260620 07:19:34.245543 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 26615
I20260620 07:19:34.251462 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 26782
I20260620 07:19:34.251686 27101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:34.251250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:45003 (local address 127.11.123.131:47103)
0620 07:19:34.251473 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:34.251560 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20260620 07:19:34.256778 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 26916
2026-06-20T07:19:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19272 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:19:34 dist-test-slave-j663 krb5kdc[27104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:19:34 dist-test-slave-j663 krb5kdc[27104](info): set up 2 sockets
Jun 20 07:19:34 dist-test-slave-j663 krb5kdc[27104](info): commencing operation
krb5kdc: starting...
W20260620 07:19:35.297501 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:19:35 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:19:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:19:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:19:35.418390 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:36513
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:35.524572 27117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:35.524793 27117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:35.524832 27117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:35.528730 27117 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:19:35.528777 27117 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:19:35.528792 27117 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:19:35.528806 27117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:19:35.528823 27117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:35.528841 27117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:35.528903 27117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:19:35.528923 27117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:19:35.533978 27117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:35.534904 27117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:35.535836 27117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:35.543771 27122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:35.543756 27125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:35.543846 27123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:35.544004 27117 server_base.cc:1034] running on GCE node
I20260620 07:19:35.544495 27117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:35.545547 27117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:35.546733 27117 hybrid_clock.cc:648] HybridClock initialized: now 1781939975546722 us; error 23 us; skew 500 ppm
Jun 20 07:19:35 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939975, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:35.549383 27117 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:19:35.550115 27117 webserver.cc:460] Webserver started at http://127.11.123.190:35253/ using document root <none> and password file <none>
I20260620 07:19:35.550570 27117 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:35.550624 27117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:35.550771 27117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:19:35.552582 27117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "70181d3e668a4047a20e9ce06ba3333c"
format_stamp: "Formatted at 2026-06-20 07:19:35 on dist-test-slave-j663"
server_key: "fcaded7a7120329ffe174b64ac0a7416"
server_key_iv: "d41577a7804a45959f02142afd826b78"
server_key_version: "encryptionkey@0"
I20260620 07:19:35.553061 27117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "70181d3e668a4047a20e9ce06ba3333c"
format_stamp: "Formatted at 2026-06-20 07:19:35 on dist-test-slave-j663"
server_key: "fcaded7a7120329ffe174b64ac0a7416"
server_key_iv: "d41577a7804a45959f02142afd826b78"
server_key_version: "encryptionkey@0"
I20260620 07:19:35.556420 27117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:19:35.558564 27132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:35.559708 27117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:19:35.559800 27117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "70181d3e668a4047a20e9ce06ba3333c"
format_stamp: "Formatted at 2026-06-20 07:19:35 on dist-test-slave-j663"
server_key: "fcaded7a7120329ffe174b64ac0a7416"
server_key_iv: "d41577a7804a45959f02142afd826b78"
server_key_version: "encryptionkey@0"
I20260620 07:19:35.559873 27117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:35.583328 27117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:35.586611 27117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:35.586758 27117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:35.593962 27117 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:36513
I20260620 07:19:35.594251 27184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:36513 every 8 connection(s)
I20260620 07:19:35.594966 27117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:19:35.597657 27185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:35.602917 27185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Bootstrap starting.
I20260620 07:19:35.604152 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 27117
I20260620 07:19:35.604252 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:19:35.604660 11758 external_mini_cluster.cc:1428] Setting key d687c7505b0a18b5d43d614e86205e3c
I20260620 07:19:35.605521 27185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:35.606398 27185 log.cc:824] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:35.608510 27185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: No bootstrap required, opened a new log
I20260620 07:19:35.612541 27185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } }
I20260620 07:19:35.612742 27185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:35.612776 27185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70181d3e668a4047a20e9ce06ba3333c, State: Initialized, Role: FOLLOWER
I20260620 07:19:35.613227 27185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } }
I20260620 07:19:35.613376 27185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:19:35.613467 27185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:19:35.613583 27185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 0 FOLLOWER]: Advancing to term 1
Jun 20 07:19:35 dist-test-slave-j663 krb5kdc[27104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:35.614603 27185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } }
I20260620 07:19:35.614971 27185 leader_election.cc:302] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70181d3e668a4047a20e9ce06ba3333c; no voters:
I20260620 07:19:35.615377 27185 leader_election.cc:288] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:19:35.615736 27191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:19:35.615942 27191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 1 LEADER]: Becoming Leader. State: Replica: 70181d3e668a4047a20e9ce06ba3333c, State: Running, Role: LEADER
I20260620 07:19:35.616264 27191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } }
I20260620 07:19:35.616641 27185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:19:35.617784 27190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70181d3e668a4047a20e9ce06ba3333c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } } }
I20260620 07:19:35.617908 27190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:35.618155 27192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70181d3e668a4047a20e9ce06ba3333c. Latest consensus state: current_term: 1 leader_uuid: "70181d3e668a4047a20e9ce06ba3333c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } } }
I20260620 07:19:35.618323 27192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:35.619514 27195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:19:35.620451 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:35.605745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.11.123.190:36513)
0620 07:19:35.606221 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:35.606241 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:35.606285 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:19:35.607054 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:35.607073 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:35.607473 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:35.607825 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:35.608446 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:35.609926 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:35.610769 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:35.611465 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:35.611681 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:35.614419 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:35.614428 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:35.614447 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:35.614484 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:35.616629 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:35.617069 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:35.617108 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:35.617116 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:35.617392 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:35.617729 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:35.617734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:35.617738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:35.618224 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:35.619119 (+ 895us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:35.619281 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 07:19:35.619666 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":154,"threads_started":1}
I20260620 07:19:35.622726 27195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:19:35.628911 27195 catalog_manager.cc:1347] Generated new cluster ID: c5cdf6d76de44e19acd2415f7cbe8e82
I20260620 07:19:35.628968 27195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:19:35.659567 27195 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:19:35.661108 27195 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:19:35.676184 27195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Generated new TSK 0
I20260620 07:19:35.677064 27195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:19:35.711762 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36513
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:35.819784 27211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:35.820022 27211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:35.820063 27211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:35.824038 27211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:35.824086 27211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:35.824165 27211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:19:35.829286 27211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:35.830216 27211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:35.831179 27211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:35.840691 27219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:35.840721 27216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:35.840811 27211 server_base.cc:1034] running on GCE node
W20260620 07:19:35.840726 27217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:35.841341 27211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:35.842069 27211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:35.843258 27211 hybrid_clock.cc:648] HybridClock initialized: now 1781939975843254 us; error 40 us; skew 500 ppm
Jun 20 07:19:35 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939975, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:35.845986 27211 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:19:35.846765 27211 webserver.cc:460] Webserver started at http://127.11.123.129:45485/ using document root <none> and password file <none>
I20260620 07:19:35.847301 27211 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:35.847388 27211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:35.847601 27211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:19:35.849570 27211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35"
format_stamp: "Formatted at 2026-06-20 07:19:35 on dist-test-slave-j663"
server_key: "18d008f87e5a1963630f4d0bcb08d319"
server_key_iv: "019beb13dd882df2b67eaad31eba1dec"
server_key_version: "encryptionkey@0"
I20260620 07:19:35.850108 27211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35"
format_stamp: "Formatted at 2026-06-20 07:19:35 on dist-test-slave-j663"
server_key: "18d008f87e5a1963630f4d0bcb08d319"
server_key_iv: "019beb13dd882df2b67eaad31eba1dec"
server_key_version: "encryptionkey@0"
I20260620 07:19:35.853586 27211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:19:35.855921 27226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:35.857146 27211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:19:35.857270 27211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35"
format_stamp: "Formatted at 2026-06-20 07:19:35 on dist-test-slave-j663"
server_key: "18d008f87e5a1963630f4d0bcb08d319"
server_key_iv: "019beb13dd882df2b67eaad31eba1dec"
server_key_version: "encryptionkey@0"
I20260620 07:19:35.857380 27211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:35.867578 27211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:35.870337 27211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:35.870532 27211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:35.871093 27211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:35.871868 27211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:35.871938 27211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:35.872006 27211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:35.872061 27211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:35.880522 27211 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:38089
I20260620 07:19:35.880553 27339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:38089 every 8 connection(s)
I20260620 07:19:35.881367 27211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:19:35.887265 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 27211
I20260620 07:19:35.887403 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:19:35.887738 11758 external_mini_cluster.cc:1428] Setting key 32fa22d25470334949256721e122f933
Jun 20 07:19:35 dist-test-slave-j663 krb5kdc[27104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939975, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:35.895432 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:35.883133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:57771 (local address 127.11.123.190:36513)
0620 07:19:35.883279 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:35.883284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:35.884092 (+ 808us) server_negotiation.cc:407] Connection header received
0620 07:19:35.885396 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:35.885399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:35.885444 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:35.885516 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:35.886614 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:35.887133 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:35.888585 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:35.888833 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:35.888938 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:35.891905 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:35.891908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:35.891910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:35.891931 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:35.893335 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:35.893828 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:35.893834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:35.893839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:35.893917 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:35.894320 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:35.894323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:35.894324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:35.894495 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:35.894587 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:35.895220 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 07:19:35.895316 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57}
I20260620 07:19:35.896559 27342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:35.883531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:57771)
0620 07:19:35.883941 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:35.883981 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:35.885204 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:35.885673 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:35.885679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:35.886008 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:35.886480 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:35.886492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:35.887260 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:35.887264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:35.888466 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:35.888477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:35.889000 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:35.889005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:35.889227 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:35.889969 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:35.890022 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:35.891736 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:35.893455 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:35.893460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:35.893473 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:35.893723 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:35.894008 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:35.894014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:35.894019 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:35.894207 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:35.894611 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:35.894621 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:35.894941 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 07:19:35.895178 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 07:19:35.895528 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":99,"threads_started":1}
I20260620 07:19:35.897795 27340 heartbeater.cc:347] Connected to a master server at 127.11.123.190:36513
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:19:35.898034 27340 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:35.898512 27340 heartbeater.cc:511] Master 127.11.123.190:36513 requested a full tablet report, sending...
I20260620 07:19:35.900038 27149 ts_manager.cc:194] Registered new tserver with Master: c04263dc1b6d4b88b073d5d0ddfb3a35 (127.11.123.129:38089)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
I20260620 07:19:35.901625 27149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:57771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:19:35.916024 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36513
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:36.024315 27345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:36.024546 27345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:36.024587 27345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:36.028581 27345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:36.028633 27345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:36.028720 27345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:19:36.033896 27345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:36.034911 27345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:36.035919 27345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:36.045125 27350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:36.045136 27345 server_base.cc:1034] running on GCE node
W20260620 07:19:36.045141 27351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:36.045185 27353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:36.045686 27345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:36.046315 27345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:36.047492 27345 hybrid_clock.cc:648] HybridClock initialized: now 1781939976047456 us; error 46 us; skew 500 ppm
Jun 20 07:19:36 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939976, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:36.050173 27345 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:19:36.050988 27345 webserver.cc:460] Webserver started at http://127.11.123.130:38031/ using document root <none> and password file <none>
I20260620 07:19:36.051533 27345 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:36.051621 27345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:36.051851 27345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:19:36.053834 27345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "b265150651a54d5f9d62064de91ddc26"
format_stamp: "Formatted at 2026-06-20 07:19:36 on dist-test-slave-j663"
server_key: "29711b8b415aa407bc513f60a4c397cf"
server_key_iv: "ae7c63287caba4ffde5fe2c944cffb17"
server_key_version: "encryptionkey@0"
I20260620 07:19:36.054370 27345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "b265150651a54d5f9d62064de91ddc26"
format_stamp: "Formatted at 2026-06-20 07:19:36 on dist-test-slave-j663"
server_key: "29711b8b415aa407bc513f60a4c397cf"
server_key_iv: "ae7c63287caba4ffde5fe2c944cffb17"
server_key_version: "encryptionkey@0"
I20260620 07:19:36.057821 27345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:19:36.060300 27360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:36.061662 27345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 07:19:36.061792 27345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "b265150651a54d5f9d62064de91ddc26"
format_stamp: "Formatted at 2026-06-20 07:19:36 on dist-test-slave-j663"
server_key: "29711b8b415aa407bc513f60a4c397cf"
server_key_iv: "ae7c63287caba4ffde5fe2c944cffb17"
server_key_version: "encryptionkey@0"
I20260620 07:19:36.061942 27345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:36.080076 27345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:36.083303 27345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:36.083508 27345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:36.084095 27345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:36.084906 27345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:36.084951 27345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:36.084990 27345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:36.085036 27345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:36.093989 27345 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:39465
I20260620 07:19:36.094061 27473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:39465 every 8 connection(s)
I20260620 07:19:36.094866 27345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:19:36.102331 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 27345
I20260620 07:19:36.102466 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:19:36.102766 11758 external_mini_cluster.cc:1428] Setting key 035b31a16b708e2d967b154a8ee9bde5
Jun 20 07:19:36 dist-test-slave-j663 krb5kdc[27104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939976, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:36.108639 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.096643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:48639 (local address 127.11.123.190:36513)
0620 07:19:36.096773 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:36.096777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:36.097601 (+ 824us) server_negotiation.cc:407] Connection header received
0620 07:19:36.098859 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:36.098862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:36.098948 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:36.099045 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:36.100153 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.100685 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.101873 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.102104 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.102216 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.104872 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:36.104876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:36.104878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:36.104914 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:36.106589 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:36.107105 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:36.107111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:36.107113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:36.107213 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:36.107569 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:36.107574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:36.107576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:36.107751 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:36.107860 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:36.108352 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 07:19:36.108470 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:19:36.109475 27476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.097024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:48639)
0620 07:19:36.097464 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:36.097503 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:36.098648 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:36.099216 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:36.099223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:36.099568 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:36.100016 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.100029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.100807 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.100811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.101749 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.101758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.102252 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.102256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.102469 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.103129 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:36.103155 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:36.104714 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:36.106715 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:36.106721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:36.106736 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:36.106999 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:36.107314 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:36.107316 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:36.107318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:36.107471 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:36.107858 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:36.107864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:36.108108 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 07:19:36.108291 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 07:19:36.108607 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":100,"threads_started":1}
I20260620 07:19:36.110680 27474 heartbeater.cc:347] Connected to a master server at 127.11.123.190:36513
I20260620 07:19:36.110936 27474 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:36.111435 27474 heartbeater.cc:511] Master 127.11.123.190:36513 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
I20260620 07:19:36.112466 27149 ts_manager.cc:194] Registered new tserver with Master: b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465)
I20260620 07:19:36.113133 27149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:48639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:19:36.127964 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36513
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:36.235222 27479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:36.235457 27479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:36.235499 27479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:36.239463 27479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:36.239518 27479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:36.239610 27479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:19:36.244681 27479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:36.245697 27479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:36.246681 27479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:19:36.256192 27479 server_base.cc:1034] running on GCE node
W20260620 07:19:36.256160 27485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:36.256142 27484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:36.256124 27487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:36.256695 27479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:36.257361 27479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:36.258549 27479 hybrid_clock.cc:648] HybridClock initialized: now 1781939976258532 us; error 26 us; skew 500 ppm
Jun 20 07:19:36 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939976, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:36.261219 27479 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:19:36.262179 27479 webserver.cc:460] Webserver started at http://127.11.123.131:45739/ using document root <none> and password file <none>
I20260620 07:19:36.262722 27479 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:36.262809 27479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:36.263031 27479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:19:36.265034 27479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "bf7520db1f0548958c30ac156f6a3b71"
format_stamp: "Formatted at 2026-06-20 07:19:36 on dist-test-slave-j663"
server_key: "e09c55822ab3d6fb6674223e5a5e27a4"
server_key_iv: "287309880c95c36037e008ee0279a687"
server_key_version: "encryptionkey@0"
I20260620 07:19:36.265661 27479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "bf7520db1f0548958c30ac156f6a3b71"
format_stamp: "Formatted at 2026-06-20 07:19:36 on dist-test-slave-j663"
server_key: "e09c55822ab3d6fb6674223e5a5e27a4"
server_key_iv: "287309880c95c36037e008ee0279a687"
server_key_version: "encryptionkey@0"
I20260620 07:19:36.269332 27479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:19:36.271667 27494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:36.272825 27479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:36.272951 27479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "bf7520db1f0548958c30ac156f6a3b71"
format_stamp: "Formatted at 2026-06-20 07:19:36 on dist-test-slave-j663"
server_key: "e09c55822ab3d6fb6674223e5a5e27a4"
server_key_iv: "287309880c95c36037e008ee0279a687"
server_key_version: "encryptionkey@0"
I20260620 07:19:36.273119 27479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:36.290584 27479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:36.293463 27479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:36.293649 27479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:36.294196 27479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:36.294966 27479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:36.295034 27479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:36.295109 27479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:36.295157 27479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:36.303763 27479 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:37943
I20260620 07:19:36.303785 27607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:37943 every 8 connection(s)
I20260620 07:19:36.304601 27479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:19:36.313794 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 27479
I20260620 07:19:36.313920 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
Jun 20 07:19:36 dist-test-slave-j663 krb5kdc[27104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939976, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:36.314248 11758 external_mini_cluster.cc:1428] Setting key cab67fa80099fcd14c5e081470740d8e
I20260620 07:19:36.318622 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.306482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:39993 (local address 127.11.123.190:36513)
0620 07:19:36.306643 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:36.306646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:36.307451 (+ 805us) server_negotiation.cc:407] Connection header received
0620 07:19:36.308683 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:36.308687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:36.308743 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:36.308850 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:36.309979 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.310534 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.311646 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.311876 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.311989 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.314734 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:36.314737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:36.314739 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:36.314762 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:36.316518 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:36.317108 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:36.317113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:36.317115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:36.317204 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:36.317538 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:36.317543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:36.317545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:36.317730 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:36.317858 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:36.318336 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 07:19:36.318453 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:19:36.319486 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.306826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:39993)
0620 07:19:36.307294 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:36.307334 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:36.308479 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:36.309018 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:36.309024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:36.309376 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:36.309830 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.309842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.310661 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.310665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.311520 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.311529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.312029 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.312033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.312194 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.312816 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:36.312832 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:36.314553 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:36.316637 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:36.316642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:36.316656 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:36.316972 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:36.317305 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:36.317308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:36.317310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:36.317446 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:36.317855 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:36.317861 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:36.318103 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 07:19:36.318282 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 07:19:36.318622 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":137,"threads_started":1}
I20260620 07:19:36.320611 27608 heartbeater.cc:347] Connected to a master server at 127.11.123.190:36513
I20260620 07:19:36.320936 27608 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:36.321453 27608 heartbeater.cc:511] Master 127.11.123.190:36513 requested a full tablet report, sending...
I20260620 07:19:36.322513 27149 ts_manager.cc:194] Registered new tserver with Master: bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943)
I20260620 07:19:36.323102 27149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:39993
I20260620 07:19:36.328032 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:19:36.336664 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.329936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.11.123.190:36513)
0620 07:19:36.330078 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:36.330084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:36.330225 (+ 141us) server_negotiation.cc:407] Connection header received
0620 07:19:36.330316 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:36.330318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:36.330353 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:36.330430 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:36.331032 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.331533 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.332351 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.332562 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.332639 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.333970 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:36.333972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:36.333975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:36.333994 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:36.335241 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:36.335608 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:36.335610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:36.335611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:36.335679 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:36.335899 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:36.335902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:36.335904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:36.336101 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:36.336200 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:36.336372 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 07:19:36.336461 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:19:36.339424 27149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:19:36.342384 27149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:19:36.355995 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.350384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:37943 (local address 127.0.0.1:55458)
0620 07:19:36.351466 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:36.351481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:36.351605 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:36.352028 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:36.352034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:36.352055 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:36.352199 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.352207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.353337 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.353340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.354592 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.354602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.355442 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.355445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.355559 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.355595 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:19:36.355736 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 07:19:36.355850 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":84,"spinlock_wait_cycles":2944,"thread_start_us":97,"threads_started":1}
I20260620 07:19:36.355997 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.350540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.11.123.131:37943)
0620 07:19:36.351322 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:36.351333 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:36.351584 (+ 251us) server_negotiation.cc:407] Connection header received
0620 07:19:36.351721 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:36.351725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:36.351866 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:36.352027 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:36.352330 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.353177 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.354714 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.355317 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.355462 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.355592 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:36.355766 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 07:19:36.355858 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":143,"threads_started":1}
I20260620 07:19:36.356197 27620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.350213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:38089 (local address 127.0.0.1:57198)
0620 07:19:36.350852 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:36.350903 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:36.351025 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:36.352029 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:36.352034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:36.352078 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:36.352206 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.352213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.353775 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.353777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.354756 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.354766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.355833 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.355840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.355933 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.355958 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:19:36.356005 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 07:19:36.356068 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":82,"spinlock_wait_cycles":3200,"thread_start_us":68,"threads_started":1}
I20260620 07:19:36.356761 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.350368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.11.123.129:38089)
0620 07:19:36.351022 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:36.351034 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:36.351550 (+ 516us) server_negotiation.cc:407] Connection header received
0620 07:19:36.351647 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:36.351655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:36.351867 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:36.352342 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:36.352506 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.353645 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.354885 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.355695 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.356311 (+ 616us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.356425 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:36.356476 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 07:19:36.356578 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":109,"threads_started":1}
I20260620 07:19:36.357328 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.351144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:39465 (local address 127.0.0.1:35960)
0620 07:19:36.351460 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:36.351473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:36.351570 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:36.352219 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:36.352222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:36.352241 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:36.352328 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.352334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.353742 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.353747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.355765 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.355777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.356837 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.356841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.356931 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.356956 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:19:36.357009 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 07:19:36.357208 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":723,"thread_start_us":91,"threads_started":1}
I20260620 07:19:36.357970 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.351286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.11.123.130:39465)
0620 07:19:36.351802 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:36.351813 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:36.351833 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:19:36.351901 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:36.351906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:36.352070 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:36.352245 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:36.352409 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.353584 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.355894 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.356693 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.357539 (+ 846us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.357664 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:36.357721 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 07:19:36.357824 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":115,"threads_started":1}
I20260620 07:19:36.359534 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0582c078ee442f972fca5438e602a4 (DEFAULT_TABLE table=test-table [id=c294ba74d90d411a8ab2a9dffd6fdc17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:36.359531 27542 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0582c078ee442f972fca5438e602a4 (DEFAULT_TABLE table=test-table [id=c294ba74d90d411a8ab2a9dffd6fdc17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:36.360635 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0582c078ee442f972fca5438e602a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:36.360639 27542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0582c078ee442f972fca5438e602a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:36.360924 27408 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0582c078ee442f972fca5438e602a4 (DEFAULT_TABLE table=test-table [id=c294ba74d90d411a8ab2a9dffd6fdc17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:36.361943 27408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0582c078ee442f972fca5438e602a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:36.367900 27626 tablet_bootstrap.cc:492] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Bootstrap starting.
I20260620 07:19:36.368305 27625 tablet_bootstrap.cc:492] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71: Bootstrap starting.
I20260620 07:19:36.368945 27627 tablet_bootstrap.cc:492] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26: Bootstrap starting.
I20260620 07:19:36.370690 27626 tablet_bootstrap.cc:654] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:36.371276 27625 tablet_bootstrap.cc:654] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:36.371313 27627 tablet_bootstrap.cc:654] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:36.371930 27626 log.cc:824] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:36.372115 27625 log.cc:824] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:36.372115 27627 log.cc:824] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:36.373809 27626 tablet_bootstrap.cc:492] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: No bootstrap required, opened a new log
I20260620 07:19:36.373808 27627 tablet_bootstrap.cc:492] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26: No bootstrap required, opened a new log
I20260620 07:19:36.374011 27627 ts_tablet_manager.cc:1397] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:19:36.374017 27626 ts_tablet_manager.cc:1397] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:19:36.375130 27625 tablet_bootstrap.cc:492] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71: No bootstrap required, opened a new log
I20260620 07:19:36.375300 27625 ts_tablet_manager.cc:1397] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 07:19:36.377398 27627 raft_consensus.cc:348] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } } peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } }
I20260620 07:19:36.377740 27627 raft_consensus.cc:374] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:36.377805 27627 raft_consensus.cc:729] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b265150651a54d5f9d62064de91ddc26, State: Initialized, Role: FOLLOWER
I20260620 07:19:36.377790 27626 raft_consensus.cc:348] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } } peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } }
I20260620 07:19:36.378074 27626 raft_consensus.cc:374] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:36.378129 27626 raft_consensus.cc:729] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04263dc1b6d4b88b073d5d0ddfb3a35, State: Initialized, Role: FOLLOWER
I20260620 07:19:36.378288 27627 consensus_queue.cc:260] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } } peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } }
I20260620 07:19:36.378571 27626 consensus_queue.cc:260] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } } peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } }
I20260620 07:19:36.378978 27474 heartbeater.cc:503] Master 127.11.123.190:36513 was elected leader, sending a full tablet report...
I20260620 07:19:36.379030 27625 raft_consensus.cc:348] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } } peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } }
I20260620 07:19:36.379266 27627 ts_tablet_manager.cc:1428] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:19:36.379266 27625 raft_consensus.cc:374] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:36.379315 27625 raft_consensus.cc:729] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7520db1f0548958c30ac156f6a3b71, State: Initialized, Role: FOLLOWER
I20260620 07:19:36.379555 27626 ts_tablet_manager.cc:1428] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:19:36.379695 27625 consensus_queue.cc:260] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } } peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } }
I20260620 07:19:36.379880 27340 heartbeater.cc:503] Master 127.11.123.190:36513 was elected leader, sending a full tablet report...
I20260620 07:19:36.380537 27625 ts_tablet_manager.cc:1428] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 07:19:36.380571 27608 heartbeater.cc:503] Master 127.11.123.190:36513 was elected leader, sending a full tablet report...
W20260620 07:19:36.382431 27341 tablet.cc:2369] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:19:36.555748 27609 tablet.cc:2369] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:19:36.596077 27475 tablet.cc:2369] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:19:36.643528 27633 raft_consensus.cc:482] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:19:36.643786 27633 raft_consensus.cc:504] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } } peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } }
I20260620 07:19:36.644990 27633 leader_election.cc:288] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04263dc1b6d4b88b073d5d0ddfb3a35 (127.11.123.129:38089), b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465)
I20260620 07:19:36.648597 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.645323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:37075 (local address 127.11.123.129:38089)
0620 07:19:36.645462 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:36.645467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:36.645481 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:19:36.645667 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:36.645673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:36.645733 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:36.645820 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:36.646126 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.646629 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.647646 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.648034 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.648173 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.648223 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:36.648408 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 07:19:36.648477 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:19:36.648595 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.645248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:38089 (local address 127.11.123.131:37075)
0620 07:19:36.645398 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:36.645416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:36.645515 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:36.645865 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:36.645869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:36.645888 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:36.646003 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.646009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.646745 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.646750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.647510 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.647518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.648164 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.648168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.648274 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.648303 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 07:19:36.648391 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 07:19:36.648462 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 07:19:36.649067 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.645534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:39465 (local address 127.11.123.131:58611)
0620 07:19:36.645921 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:36.645932 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:36.646049 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:36.646303 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:36.646306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:36.646321 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:36.646451 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.646456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.647156 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.647159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.647981 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:36.647990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.648626 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.648629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:36.648734 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.648765 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:19:36.648870 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 07:19:36.648930 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":133,"threads_started":1}
I20260620 07:19:36.649058 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.645673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:58611 (local address 127.11.123.130:39465)
0620 07:19:36.645797 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:36.645803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:36.646005 (+ 202us) server_negotiation.cc:407] Connection header received
0620 07:19:36.646159 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:36.646163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:36.646211 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:36.646295 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:36.646566 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.647041 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.648110 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.648490 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.648626 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.648674 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:36.648889 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 07:19:36.648958 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:19:36.649278 27294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0582c078ee442f972fca5438e602a4" candidate_uuid: "bf7520db1f0548958c30ac156f6a3b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" is_pre_election: true
I20260620 07:19:36.649633 27294 raft_consensus.cc:2454] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7520db1f0548958c30ac156f6a3b71 in term 0.
I20260620 07:19:36.649781 27428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0582c078ee442f972fca5438e602a4" candidate_uuid: "bf7520db1f0548958c30ac156f6a3b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b265150651a54d5f9d62064de91ddc26" is_pre_election: true
I20260620 07:19:36.650043 27428 raft_consensus.cc:2454] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7520db1f0548958c30ac156f6a3b71 in term 0.
I20260620 07:19:36.650107 27495 leader_election.cc:302] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf7520db1f0548958c30ac156f6a3b71, c04263dc1b6d4b88b073d5d0ddfb3a35; no voters:
I20260620 07:19:36.650381 27633 raft_consensus.cc:2781] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:19:36.650497 27633 raft_consensus.cc:482] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:19:36.650557 27633 raft_consensus.cc:3037] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:36.651548 27633 raft_consensus.cc:504] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } } peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } }
I20260620 07:19:36.651885 27633 leader_election.cc:288] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [CANDIDATE]: Term 1 election: Requested vote from peers c04263dc1b6d4b88b073d5d0ddfb3a35 (127.11.123.129:38089), b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465)
I20260620 07:19:36.652261 27294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0582c078ee442f972fca5438e602a4" candidate_uuid: "bf7520db1f0548958c30ac156f6a3b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35"
I20260620 07:19:36.652294 27428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0582c078ee442f972fca5438e602a4" candidate_uuid: "bf7520db1f0548958c30ac156f6a3b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b265150651a54d5f9d62064de91ddc26"
I20260620 07:19:36.652386 27294 raft_consensus.cc:3037] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:36.652390 27428 raft_consensus.cc:3037] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:36.653635 27428 raft_consensus.cc:2454] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7520db1f0548958c30ac156f6a3b71 in term 1.
I20260620 07:19:36.653635 27294 raft_consensus.cc:2454] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7520db1f0548958c30ac156f6a3b71 in term 1.
I20260620 07:19:36.653914 27495 leader_election.cc:302] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf7520db1f0548958c30ac156f6a3b71, c04263dc1b6d4b88b073d5d0ddfb3a35; no voters:
I20260620 07:19:36.654069 27633 raft_consensus.cc:2781] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:19:36.654290 27633 raft_consensus.cc:686] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 1 LEADER]: Becoming Leader. State: Replica: bf7520db1f0548958c30ac156f6a3b71, State: Running, Role: LEADER
I20260620 07:19:36.654608 27633 consensus_queue.cc:237] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } } peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } }
I20260620 07:19:36.657707 27147 catalog_manager.cc:5495] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131). New cstate: current_term: 1 leader_uuid: "bf7520db1f0548958c30ac156f6a3b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } health_report { overall_health: UNKNOWN } } }
I20260620 07:19:36.715449 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.712273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.11.123.131:37943)
0620 07:19:36.712417 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:36.712420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:36.712434 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:19:36.712506 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:36.712508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:36.712570 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:36.712661 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:19:36.712933 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.713438 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.714226 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:36.714426 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:36.714500 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:36.714751 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:19:36.715151 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:19:36.715239 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:36.715298 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 07:19:36.715344 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:19:36.721232 27294 raft_consensus.cc:1261] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Refusing update from remote peer bf7520db1f0548958c30ac156f6a3b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:19:36.721232 27428 raft_consensus.cc:1261] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26 [term 1 FOLLOWER]: Refusing update from remote peer bf7520db1f0548958c30ac156f6a3b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:19:36.721935 27633 consensus_queue.cc:1035] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:19:36.722119 27635 consensus_queue.cc:1035] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:19:36.729455 27640 mvcc.cc:205] Tried to move back new op lower bound from 7298826144642510848 to 7298826144380153856. Current Snapshot: MvccSnapshot[applied={T|T < 7298826144642510848}]
I20260620 07:19:36.737496 27147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:19:36.737669 27147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:19:36.740010 27147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Sending DeleteTablet for 3 replicas of tablet 2e0582c078ee442f972fca5438e602a4
I20260620 07:19:36.740742 27274 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0582c078ee442f972fca5438e602a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:36 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:57198
I20260620 07:19:36.740830 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 27211
I20260620 07:19:36.740818 27408 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0582c078ee442f972fca5438e602a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:36 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:35960
I20260620 07:19:36.741734 27542 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0582c078ee442f972fca5438e602a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:36 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:55458
I20260620 07:19:36.742694 27648 tablet_replica.cc:330] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26: stopping tablet replica
I20260620 07:19:36.742980 27648 raft_consensus.cc:2229] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:19:36.743294 27648 raft_consensus.cc:2258] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:19:36.743656 27649 tablet_replica.cc:330] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71: stopping tablet replica
I20260620 07:19:36.743925 27649 raft_consensus.cc:2229] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:19:36.744233 27649 raft_consensus.cc:2258] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:19:36.744915 27648 ts_tablet_manager.cc:1905] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:19:36.746315 27649 ts_tablet_manager.cc:1905] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:19:36.748353 27648 ts_tablet_manager.cc:1918] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:19:36.748459 27648 log.cc:1196] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/2e0582c078ee442f972fca5438e602a4
I20260620 07:19:36.748811 27648 ts_tablet_manager.cc:1939] T 2e0582c078ee442f972fca5438e602a4 P b265150651a54d5f9d62064de91ddc26: Deleting consensus metadata
I20260620 07:19:36.749150 27649 ts_tablet_manager.cc:1918] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 07:19:36.749219 27133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:38089 (error 108)
I20260620 07:19:36.749333 27649 log.cc:1196] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/2e0582c078ee442f972fca5438e602a4
I20260620 07:19:36.749711 27649 ts_tablet_manager.cc:1939] T 2e0582c078ee442f972fca5438e602a4 P bf7520db1f0548958c30ac156f6a3b71: Deleting consensus metadata
I20260620 07:19:36.749984 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 27345
I20260620 07:19:36.750509 27133 catalog_manager.cc:4841] TS b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465): tablet 2e0582c078ee442f972fca5438e602a4 (table test-table [id=c294ba74d90d411a8ab2a9dffd6fdc17]) successfully deleted
I20260620 07:19:36.750988 27136 catalog_manager.cc:4841] TS bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943): tablet 2e0582c078ee442f972fca5438e602a4 (table test-table [id=c294ba74d90d411a8ab2a9dffd6fdc17]) successfully deleted
I20260620 07:19:36.752384 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:36.752070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:38089 (local address 127.0.0.1:57210)
0620 07:19:36.752208 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:36.752284 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 07:19:36.752581 27133 catalog_manager.cc:4584] TS c04263dc1b6d4b88b073d5d0ddfb3a35 (127.11.123.129:38089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e0582c078ee442f972fca5438e602a4: Network error: Client connection negotiation failed: client connection to 127.11.123.129:38089: connect: Connection refused (error 111)
I20260620 07:19:36.756249 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 27479
I20260620 07:19:36.760963 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 27117
I20260620 07:19:56.767789 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:36513
--webserver_interface=127.11.123.190
--webserver_port=35253
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:19:56.879884 27651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:56.880122 27651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:56.880162 27651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:56.884091 27651 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:19:56.884155 27651 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:19:56.884171 27651 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:19:56.884184 27651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:19:56.884202 27651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:56.884245 27651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:56.884277 27651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:19:56.884297 27651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:19:56.889496 27651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:56.890515 27651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:56.891534 27651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:56.904572 27657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:56.904560 27658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:56.904557 27660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:56.904623 27651 server_base.cc:1034] running on GCE node
I20260620 07:19:56.905532 27651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:56.906682 27651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:56.907843 27651 hybrid_clock.cc:648] HybridClock initialized: now 1781939996907832 us; error 24 us; skew 500 ppm
Jun 20 07:19:56 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939996, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:56.910706 27651 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:19:56.911535 27651 webserver.cc:460] Webserver started at http://127.11.123.190:35253/ using document root <none> and password file <none>
I20260620 07:19:56.912060 27651 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:56.912129 27651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:56.915941 27651 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:56.920238 27667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:56.921864 27651 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 07:19:56.921990 27651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "70181d3e668a4047a20e9ce06ba3333c"
format_stamp: "Formatted at 2026-06-20 07:19:35 on dist-test-slave-j663"
server_key: "fcaded7a7120329ffe174b64ac0a7416"
server_key_iv: "d41577a7804a45959f02142afd826b78"
server_key_version: "encryptionkey@0"
I20260620 07:19:56.922756 27651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:56.938700 27651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:56.941388 27651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:56.941584 27651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:56.948463 27651 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:36513
I20260620 07:19:56.948762 27719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:36513 every 8 connection(s)
I20260620 07:19:56.949536 27651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:19:56.953675 27720 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:19:56.954207 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 27651
I20260620 07:19:56.954682 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:38089
--local_ip_for_outbound_sockets=127.11.123.129
--tserver_master_addrs=127.11.123.190:36513
--webserver_port=45485
--webserver_interface=127.11.123.129
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:19:56.955797 27720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Bootstrap starting.
I20260620 07:19:56.978937 27720 log.cc:824] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:56.987703 27720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:19:56.988088 27720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Bootstrap complete.
I20260620 07:19:56.992204 27720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } }
I20260620 07:19:56.992594 27720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70181d3e668a4047a20e9ce06ba3333c, State: Initialized, Role: FOLLOWER
I20260620 07:19:56.993062 27720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } }
I20260620 07:19:56.993235 27720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:19:56.993327 27720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:19:56.993454 27720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:19:56.995301 27720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } }
I20260620 07:19:56.995643 27720 leader_election.cc:302] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70181d3e668a4047a20e9ce06ba3333c; no voters:
I20260620 07:19:56.996081 27720 leader_election.cc:288] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:19:56.996191 27724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:19:56.996443 27724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 2 LEADER]: Becoming Leader. State: Replica: 70181d3e668a4047a20e9ce06ba3333c, State: Running, Role: LEADER
I20260620 07:19:56.996781 27724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } }
I20260620 07:19:56.997414 27720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:19:56.997784 27724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "70181d3e668a4047a20e9ce06ba3333c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } } }
I20260620 07:19:56.997995 27724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:56.998246 27724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70181d3e668a4047a20e9ce06ba3333c. Latest consensus state: current_term: 2 leader_uuid: "70181d3e668a4047a20e9ce06ba3333c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } } }
I20260620 07:19:56.998412 27724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: This master's current role is: LEADER
I20260620 07:19:56.998720 27728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:19:57.003723 27728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:19:57.004522 27728 catalog_manager.cc:1259] Loaded cluster ID: c5cdf6d76de44e19acd2415f7cbe8e82
I20260620 07:19:57.004601 27728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:19:57.006436 27728 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:19:57.007027 27728 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Loaded TSK: 0
I20260620 07:19:57.022979 27728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Generated new TSK 1
I20260620 07:19:57.023216 27728 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:19:57.099397 27722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:57.099763 27722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:57.099851 27722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:57.105621 27722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:57.105691 27722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:57.105775 27722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:19:57.113070 27722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:38089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=45485
--tserver_master_addrs=127.11.123.190:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:57.114287 27722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:57.115500 27722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:57.125707 27748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:57.125723 27750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:57.125735 27747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:57.126281 27722 server_base.cc:1034] running on GCE node
I20260620 07:19:57.126722 27722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:57.127477 27722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:57.128671 27722 hybrid_clock.cc:648] HybridClock initialized: now 1781939997128652 us; error 36 us; skew 500 ppm
Jun 20 07:19:57 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939997, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:57.132306 27722 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:19:57.133323 27722 webserver.cc:460] Webserver started at http://127.11.123.129:45485/ using document root <none> and password file <none>
I20260620 07:19:57.133921 27722 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:57.133997 27722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:57.137941 27722 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:19:57.140146 27757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:57.141543 27722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:19:57.141676 27722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35"
format_stamp: "Formatted at 2026-06-20 07:19:35 on dist-test-slave-j663"
server_key: "18d008f87e5a1963630f4d0bcb08d319"
server_key_iv: "019beb13dd882df2b67eaad31eba1dec"
server_key_version: "encryptionkey@0"
I20260620 07:19:57.142228 27722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:57.160207 27722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:57.163472 27722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:57.163681 27722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:57.164346 27722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:57.165526 27764 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 07:19:57.170444 27722 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:19:57.170521 27722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 07:19:57.170601 27722 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:19:57.172243 27722 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:19:57.172314 27722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 07:19:57.172580 27764 tablet_bootstrap.cc:492] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Bootstrap starting.
I20260620 07:19:57.183079 27722 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:38089
I20260620 07:19:57.183995 27722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:19:57.184255 27871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:38089 every 8 connection(s)
I20260620 07:19:57.192116 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 27722
I20260620 07:19:57.192622 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:39465
--local_ip_for_outbound_sockets=127.11.123.130
--tserver_master_addrs=127.11.123.190:36513
--webserver_port=38031
--webserver_interface=127.11.123.130
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:19:57.195309 27764 log.cc:824] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:19:57 dist-test-slave-j663 krb5kdc[27104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939997, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:57.202697 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.186139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:49827 (local address 127.11.123.190:36513)
0620 07:19:57.187100 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:57.187117 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:57.187499 (+ 382us) server_negotiation.cc:407] Connection header received
0620 07:19:57.188578 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:57.188588 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:57.188875 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:57.189229 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:57.190345 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.191283 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.192805 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.193621 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.193799 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.197275 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:57.197284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:57.197304 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:57.197338 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:57.199459 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:57.200066 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:57.200071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:57.200077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:57.200246 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:57.200580 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:57.200585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:57.200587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:57.200917 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:57.201059 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:57.201606 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 07:19:57.201849 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":111,"threads_started":1}
I20260620 07:19:57.202713 27875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.186160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:49827)
0620 07:19:57.187323 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:57.187375 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:57.188171 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:57.189294 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:57.189302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:57.189722 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:57.190164 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.190178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.191458 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.191462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.192647 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.192659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.193998 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.194004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.194245 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.194961 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:57.194987 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:57.196985 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:57.199590 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:57.199596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:57.199609 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:57.199949 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:57.200336 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:57.200340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:57.200342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:57.200481 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:57.201035 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:57.201041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:57.201306 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 07:19:57.201494 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 07:19:57.201818 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":153,"threads_started":1}
I20260620 07:19:57.204015 27764 tablet_bootstrap.cc:492] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:19:57.204469 27764 tablet_bootstrap.cc:492] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Bootstrap complete.
I20260620 07:19:57.204967 27872 heartbeater.cc:347] Connected to a master server at 127.11.123.190:36513
I20260620 07:19:57.205217 27872 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:57.205693 27872 heartbeater.cc:511] Master 127.11.123.190:36513 requested a full tablet report, sending...
I20260620 07:19:57.205993 27764 ts_tablet_manager.cc:1397] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.009s
I20260620 07:19:57.208377 27684 ts_manager.cc:194] Registered new tserver with Master: c04263dc1b6d4b88b073d5d0ddfb3a35 (127.11.123.129:38089)
I20260620 07:19:57.209496 27764 raft_consensus.cc:348] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } } peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } }
I20260620 07:19:57.209991 27764 raft_consensus.cc:729] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04263dc1b6d4b88b073d5d0ddfb3a35, State: Initialized, Role: FOLLOWER
I20260620 07:19:57.210471 27764 consensus_queue.cc:260] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } } peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } }
I20260620 07:19:57.211414 27764 ts_tablet_manager.cc:1428] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260620 07:19:57.212102 27684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:49827
I20260620 07:19:57.214604 27872 heartbeater.cc:503] Master 127.11.123.190:36513 was elected leader, sending a full tablet report...
Jun 20 07:19:57 dist-test-slave-j663 krb5kdc[27104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939996, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.129@KRBTEST.COM
I20260620 07:19:57.222422 27880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.212131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55302 (local address 127.11.123.129:38089)
0620 07:19:57.212408 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:57.212418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:57.212438 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:19:57.212499 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:57.212505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:57.212681 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:57.213551 (+ 870us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:57.213808 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.214793 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.215874 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.216364 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.216486 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.218778 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:57.218786 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:57.218796 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:57.218815 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:57.220489 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:57.220973 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:57.220978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:57.220980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:57.221064 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:57.221430 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:57.221435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:57.221437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:57.221651 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:57.221797 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:57.222131 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 07:19:57.222267 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":117,"threads_started":1}
I20260620 07:19:57.222421 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.211415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:38089 (local address 127.0.0.1:55302)
0620 07:19:57.211855 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:57.211900 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:57.212024 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:57.212855 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:57.212861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:57.213280 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:57.213405 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.213412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.214946 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.214953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.215738 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.215748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.216489 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.216493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.216610 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.217191 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:57.217203 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:57.218620 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:57.220603 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:57.220612 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:57.220617 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:57.220847 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:57.221225 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:57.221229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:57.221231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:57.221327 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:57.221771 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:57.221777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:57.221907 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 07:19:57.222049 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 07:19:57.222245 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":92,"threads_started":1}
I20260620 07:19:57.223376 27806 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0582c078ee442f972fca5438e602a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:36 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:55302
I20260620 07:19:57.224176 27881 tablet_replica.cc:330] stopping tablet replica
I20260620 07:19:57.224474 27881 raft_consensus.cc:2229] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:19:57.224705 27881 raft_consensus.cc:2258] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:19:57.225728 27881 ts_tablet_manager.cc:1905] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:19:57.230077 27881 ts_tablet_manager.cc:1918] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:19:57.230176 27881 log.cc:1196] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal/wals/2e0582c078ee442f972fca5438e602a4
I20260620 07:19:57.230435 27881 ts_tablet_manager.cc:1939] T 2e0582c078ee442f972fca5438e602a4 P c04263dc1b6d4b88b073d5d0ddfb3a35: Deleting consensus metadata
I20260620 07:19:57.232317 27668 catalog_manager.cc:4841] TS c04263dc1b6d4b88b073d5d0ddfb3a35 (127.11.123.129:38089): tablet 2e0582c078ee442f972fca5438e602a4 (table test-table [id=c294ba74d90d411a8ab2a9dffd6fdc17]) successfully deleted
W20260620 07:19:57.314523 27876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:57.314757 27876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:57.314834 27876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:57.318990 27876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:57.319046 27876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:57.319159 27876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:19:57.324308 27876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:39465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=38031
--tserver_master_addrs=127.11.123.190:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:57.325376 27876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:57.326354 27876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:57.336283 27886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:57.336450 27889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:57.336346 27876 server_base.cc:1034] running on GCE node
W20260620 07:19:57.336273 27887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:57.336927 27876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:57.337585 27876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:57.338768 27876 hybrid_clock.cc:648] HybridClock initialized: now 1781939997338756 us; error 24 us; skew 500 ppm
Jun 20 07:19:57 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939997, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:57.341358 27876 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:19:57.342198 27876 webserver.cc:460] Webserver started at http://127.11.123.130:38031/ using document root <none> and password file <none>
I20260620 07:19:57.342736 27876 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:57.342854 27876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:57.346678 27876 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:19:57.348692 27896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:57.349902 27876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:57.350031 27876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "b265150651a54d5f9d62064de91ddc26"
format_stamp: "Formatted at 2026-06-20 07:19:36 on dist-test-slave-j663"
server_key: "29711b8b415aa407bc513f60a4c397cf"
server_key_iv: "ae7c63287caba4ffde5fe2c944cffb17"
server_key_version: "encryptionkey@0"
I20260620 07:19:57.350553 27876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:57.365532 27876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:57.368340 27876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:57.368533 27876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:57.369136 27876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:57.369917 27876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:57.369995 27876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:57.370085 27876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:57.370138 27876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:57.379477 27876 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:39465
I20260620 07:19:57.379518 28009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:39465 every 8 connection(s)
I20260620 07:19:57.380403 27876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:19:57.382985 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 27876
I20260620 07:19:57.383453 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:37943
--local_ip_for_outbound_sockets=127.11.123.131
--tserver_master_addrs=127.11.123.190:36513
--webserver_port=45739
--webserver_interface=127.11.123.131
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:19:57 dist-test-slave-j663 krb5kdc[27104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939997, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:57.393827 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.382173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:51203 (local address 127.11.123.190:36513)
0620 07:19:57.382331 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:57.382336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:57.383345 (+ 1009us) server_negotiation.cc:407] Connection header received
0620 07:19:57.384672 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:57.384675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:57.384733 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:57.384815 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:57.386016 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.386550 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.387656 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.387895 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.388002 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.390418 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:57.390421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:57.390423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:57.390445 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:57.391854 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:57.392351 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:57.392354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:57.392355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:57.392427 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:57.392745 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:57.392750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:57.392752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:57.392919 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:57.393023 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:57.393535 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 07:19:57.393661 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:19:57.394610 28012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.382624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:51203)
0620 07:19:57.383195 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:57.383236 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:57.384464 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:57.384986 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:57.384993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:57.385397 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:57.385861 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.385875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.386675 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.386679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.387528 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.387536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.388048 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.388052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.388212 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.388821 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:57.388844 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:57.390251 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:57.391968 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:57.391974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:57.391986 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:57.392232 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:57.392513 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:57.392517 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:57.392519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:57.392651 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:57.393019 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:57.393025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:57.393305 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 07:19:57.393483 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 07:19:57.393777 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":160,"threads_started":1}
I20260620 07:19:57.395802 28010 heartbeater.cc:347] Connected to a master server at 127.11.123.190:36513
I20260620 07:19:57.396032 28010 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:57.396479 28010 heartbeater.cc:511] Master 127.11.123.190:36513 requested a full tablet report, sending...
I20260620 07:19:57.398391 27684 ts_manager.cc:194] Registered new tserver with Master: b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465)
I20260620 07:19:57.399147 27684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:51203
W20260620 07:19:57.496598 28013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:19:57.496861 28013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:19:57.496938 28013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:19:57.501482 28013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:19:57.501555 28013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:19:57.501643 28013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:19:57.506908 28013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:37943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=45739
--tserver_master_addrs=127.11.123.190:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:19:57.507895 28013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:19:57.508883 28013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:19:57.518143 28018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:57.518153 28021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:19:57.518153 28019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:19:57.518203 28013 server_base.cc:1034] running on GCE node
I20260620 07:19:57.518678 28013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:19:57.519330 28013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:19:57.520488 28013 hybrid_clock.cc:648] HybridClock initialized: now 1781939997520458 us; error 41 us; skew 500 ppm
Jun 20 07:19:57 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939997, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:19:57.523269 28013 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:19:57.524137 28013 webserver.cc:460] Webserver started at http://127.11.123.131:45739/ using document root <none> and password file <none>
I20260620 07:19:57.524624 28013 fs_manager.cc:359] Metadata directory not provided
I20260620 07:19:57.524691 28013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:19:57.528620 28013 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:19:57.530712 28028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:57.531848 28013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:19:57.531962 28013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "bf7520db1f0548958c30ac156f6a3b71"
format_stamp: "Formatted at 2026-06-20 07:19:36 on dist-test-slave-j663"
server_key: "e09c55822ab3d6fb6674223e5a5e27a4"
server_key_iv: "287309880c95c36037e008ee0279a687"
server_key_version: "encryptionkey@0"
I20260620 07:19:57.532552 28013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:19:57.575225 28013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:19:57.578202 28013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:19:57.578356 28013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:19:57.578963 28013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:19:57.579734 28013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:19:57.579782 28013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:57.579819 28013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:19:57.579835 28013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:19:57.588454 28013 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:37943
I20260620 07:19:57.588491 28141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:37943 every 8 connection(s)
I20260620 07:19:57.589408 28013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:19:57.589620 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 28013
I20260620 07:19:57.598984 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.590325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.11.123.190:36513)
0620 07:19:57.590467 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:57.590472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:57.590701 (+ 229us) server_negotiation.cc:407] Connection header received
0620 07:19:57.590796 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:57.590799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:57.590848 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:57.590938 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:57.591650 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.592404 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.593708 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.593930 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.594017 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.595793 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:57.595796 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:57.595798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:57.595820 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:57.597295 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:57.597763 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:57.597766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:57.597767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:57.597847 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:57.598135 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:57.598138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:57.598139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:57.598352 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:57.598549 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:57.598756 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 07:19:57.598846 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 07:19:57 dist-test-slave-j663 krb5kdc[27104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939997, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:19:57.604946 28145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.591753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:53365 (local address 127.11.123.190:36513)
0620 07:19:57.591965 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:57.591971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:57.592909 (+ 938us) server_negotiation.cc:407] Connection header received
0620 07:19:57.594564 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:57.594568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:57.594627 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:57.594720 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:57.596142 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.596646 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.598109 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.598443 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.598533 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.601138 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:57.601141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:57.601143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:57.601165 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:57.602818 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:57.603390 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:57.603398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:57.603402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:57.603515 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:57.603835 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:57.603839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:57.603840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:57.604064 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:57.604178 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:57.604619 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 07:19:57.604743 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
I20260620 07:19:57.605592 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.592241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:53365)
0620 07:19:57.592740 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:57.592794 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:57.594230 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:57.594928 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:57.594941 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:57.595440 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:19:57.596004 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.596017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.596772 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.596776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.597979 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.597993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.598584 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.598588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.598793 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.599339 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:19:57.599360 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:19:57.600962 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:19:57.602932 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:57.602937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:57.602950 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:57.603266 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:57.603606 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:19:57.603609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:19:57.603610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:19:57.603742 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:19:57.604171 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:19:57.604176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:19:57.604387 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 07:19:57.604533 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:19:57.604756 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":127,"threads_started":1}
I20260620 07:19:57.606739 28142 heartbeater.cc:347] Connected to a master server at 127.11.123.190:36513
I20260620 07:19:57.606945 28142 heartbeater.cc:464] Registering TS with master...
I20260620 07:19:57.607367 28142 heartbeater.cc:511] Master 127.11.123.190:36513 requested a full tablet report, sending...
I20260620 07:19:57.608335 27684 ts_manager.cc:194] Registered new tserver with Master: bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943)
I20260620 07:19:57.608877 27684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:53365
I20260620 07:19:57.612298 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:19:57.615976 28145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.612899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.11.123.190:36513)
0620 07:19:57.613015 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:57.613018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:57.613218 (+ 200us) server_negotiation.cc:407] Connection header received
0620 07:19:57.613356 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:57.613358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:57.613449 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:57.613525 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:19:57.613764 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.614231 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.615017 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.615224 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.615301 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.615602 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:19:57.615713 (+ 111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 07:19:57.615894 (+ 181us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54614: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 07:19:57.616053 28145 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54614: token expired
W20260620 07:19:57.616396 27613 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.123.190:36513)
I20260620 07:19:57.623013 28145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.616732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.11.123.190:36513)
0620 07:19:57.616839 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:57.616842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:57.616852 (+ 10us) server_negotiation.cc:407] Connection header received
0620 07:19:57.616938 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:57.616940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:57.616971 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:57.617027 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:19:57.617667 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.618106 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.618829 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.619021 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.619096 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.620403 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:19:57.620406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:19:57.620407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:19:57.620423 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:19:57.621710 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:57.622083 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:57.622085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:57.622086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:57.622155 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:19:57.622424 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:19:57.622426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:19:57.622427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:19:57.622566 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:19:57.622661 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:57.622839 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 07:19:57.622917 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:19:57.625227 27684 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:19:57.628360 27684 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:19:57.636956 27806 tablet_service.cc:1467] Processing CreateTablet for tablet f64bc93764804346a7c7729f9c3da4bc (DEFAULT_TABLE table=test-table [id=ef7b029546aa4716bd9ccf948a199f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:57.637701 27806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64bc93764804346a7c7729f9c3da4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:57.640671 28152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.636000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:39465 (local address 127.0.0.1:54822)
0620 07:19:57.636256 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:57.636270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:57.636397 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:57.637288 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:57.637291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:57.637308 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:57.637422 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.637429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.638600 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.638605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.639372 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.639380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.640259 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.640263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.640352 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.640391 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 07:19:57.640476 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:19:57.640545 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
I20260620 07:19:57.640739 28153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.636441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.11.123.130:39465)
0620 07:19:57.636811 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:57.636822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:57.636843 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:19:57.636917 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:57.636925 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:57.637141 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:57.637296 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:57.637578 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.638491 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.639495 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.640134 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.640270 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.640397 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:57.640503 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 07:19:57.640591 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":145,"threads_started":1}
I20260620 07:19:57.640892 27764 tablet_bootstrap.cc:492] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: Bootstrap starting.
I20260620 07:19:57.640924 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.635282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:37943 (local address 127.0.0.1:38864)
0620 07:19:57.635425 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:57.635438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:57.635510 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:57.636459 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:57.636462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:57.636479 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:57.636573 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.636577 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.638168 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.638172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.639391 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.639403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.640508 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.640511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.640626 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.640661 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:19:57.640712 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 07:19:57.640791 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 07:19:57.641784 27764 tablet_bootstrap.cc:654] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:57.641955 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.635474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.11.123.131:37943)
0620 07:19:57.635901 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:57.635915 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:57.635937 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:19:57.636008 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:57.636011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:57.636263 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:57.636740 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:57.636896 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.638050 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.639544 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.640366 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.641568 (+ 1202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.641681 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:57.641720 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 07:19:57.641785 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":148,"threads_started":1}
I20260620 07:19:57.643169 27764 tablet_bootstrap.cc:492] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: No bootstrap required, opened a new log
I20260620 07:19:57.643275 27764 ts_tablet_manager.cc:1397] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:19:57.643653 27764 raft_consensus.cc:348] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:19:57.643770 27764 raft_consensus.cc:374] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:57.643817 27764 raft_consensus.cc:729] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04263dc1b6d4b88b073d5d0ddfb3a35, State: Initialized, Role: FOLLOWER
I20260620 07:19:57.643946 27764 consensus_queue.cc:260] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:19:57.644348 27764 ts_tablet_manager.cc:1428] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:19:57.644164 27944 tablet_service.cc:1467] Processing CreateTablet for tablet f64bc93764804346a7c7729f9c3da4bc (DEFAULT_TABLE table=test-table [id=ef7b029546aa4716bd9ccf948a199f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:57.644555 28076 tablet_service.cc:1467] Processing CreateTablet for tablet f64bc93764804346a7c7729f9c3da4bc (DEFAULT_TABLE table=test-table [id=ef7b029546aa4716bd9ccf948a199f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:19:57.645337 27944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64bc93764804346a7c7729f9c3da4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:57.645541 28076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64bc93764804346a7c7729f9c3da4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:19:57.653635 28155 tablet_bootstrap.cc:492] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: Bootstrap starting.
I20260620 07:19:57.656152 28155 tablet_bootstrap.cc:654] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:57.656275 28156 tablet_bootstrap.cc:492] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71: Bootstrap starting.
I20260620 07:19:57.657117 28155 log.cc:824] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:57.658833 28156 tablet_bootstrap.cc:654] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71: Neither blocks nor log segments found. Creating new log.
I20260620 07:19:57.659253 28155 tablet_bootstrap.cc:492] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: No bootstrap required, opened a new log
I20260620 07:19:57.659442 28155 ts_tablet_manager.cc:1397] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:19:57.659679 28156 log.cc:824] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71: Log is configured to *not* fsync() on all Append() calls
I20260620 07:19:57.661624 28156 tablet_bootstrap.cc:492] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71: No bootstrap required, opened a new log
I20260620 07:19:57.661805 28156 ts_tablet_manager.cc:1397] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 07:19:57.663339 28155 raft_consensus.cc:348] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:19:57.663622 28155 raft_consensus.cc:374] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:57.663695 28155 raft_consensus.cc:729] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b265150651a54d5f9d62064de91ddc26, State: Initialized, Role: FOLLOWER
I20260620 07:19:57.664142 28155 consensus_queue.cc:260] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:19:57.664762 28010 heartbeater.cc:503] Master 127.11.123.190:36513 was elected leader, sending a full tablet report...
I20260620 07:19:57.665187 28155 ts_tablet_manager.cc:1428] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 07:19:57.665457 28156 raft_consensus.cc:348] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:19:57.665683 28156 raft_consensus.cc:374] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:19:57.665763 28156 raft_consensus.cc:729] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7520db1f0548958c30ac156f6a3b71, State: Initialized, Role: FOLLOWER
I20260620 07:19:57.666180 28156 consensus_queue.cc:260] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:19:57.667081 28156 ts_tablet_manager.cc:1428] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:19:57.667254 28142 heartbeater.cc:503] Master 127.11.123.190:36513 was elected leader, sending a full tablet report...
W20260620 07:19:57.685477 27873 tablet.cc:2369] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:19:57.763832 28160 raft_consensus.cc:482] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:19:57.764063 28160 raft_consensus.cc:504] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:19:57.765260 28160 leader_election.cc:288] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465), c04263dc1b6d4b88b073d5d0ddfb3a35 (127.11.123.129:38089)
I20260620 07:19:57.769690 28153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.765735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:46599 (local address 127.11.123.130:39465)
0620 07:19:57.765880 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:57.765886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:57.765903 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:19:57.766101 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:57.766107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:57.766168 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:57.766244 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:57.766653 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.767343 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.768511 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.769186 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.769278 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.769318 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:57.769543 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 07:19:57.769594 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:19:57.769726 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.765693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:39465 (local address 127.11.123.131:46599)
0620 07:19:57.765844 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:57.765861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:57.765954 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:57.766371 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:57.766374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:57.766397 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:57.766516 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.766523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.767456 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.767459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.768383 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.768391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.769307 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.769311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.769418 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.769442 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:19:57.769519 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 07:19:57.769586 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 07:19:57.770390 28161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.765980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:38089 (local address 127.11.123.131:59847)
0620 07:19:57.766456 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:57.766473 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:19:57.766569 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:19:57.766902 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:19:57.766905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:19:57.766924 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:19:57.767031 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.767037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.767951 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.767954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.768910 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:19:57.768920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.769981 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.769985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:19:57.770089 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.770107 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 07:19:57.770202 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 07:19:57.770265 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":136,"threads_started":1}
I20260620 07:19:57.770428 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.766096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:59847 (local address 127.11.123.129:38089)
0620 07:19:57.766534 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:57.766539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:57.766554 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:19:57.766716 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:57.766718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:57.766772 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:57.766857 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:19:57.767141 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.767838 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.769038 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.769726 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.769841 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.770004 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:57.770223 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 07:19:57.770297 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":96,"threads_started":1}
I20260620 07:19:57.770375 27964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64bc93764804346a7c7729f9c3da4bc" candidate_uuid: "bf7520db1f0548958c30ac156f6a3b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b265150651a54d5f9d62064de91ddc26" is_pre_election: true
I20260620 07:19:57.770673 27964 raft_consensus.cc:2454] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7520db1f0548958c30ac156f6a3b71 in term 0.
I20260620 07:19:57.770955 27826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64bc93764804346a7c7729f9c3da4bc" candidate_uuid: "bf7520db1f0548958c30ac156f6a3b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" is_pre_election: true
I20260620 07:19:57.771173 28029 leader_election.cc:302] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b265150651a54d5f9d62064de91ddc26, bf7520db1f0548958c30ac156f6a3b71; no voters:
I20260620 07:19:57.771217 27826 raft_consensus.cc:2454] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7520db1f0548958c30ac156f6a3b71 in term 0.
I20260620 07:19:57.771436 28160 raft_consensus.cc:2781] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:19:57.771528 28160 raft_consensus.cc:482] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:19:57.771612 28160 raft_consensus.cc:3037] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:57.772650 28160 raft_consensus.cc:504] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:19:57.772976 28160 leader_election.cc:288] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [CANDIDATE]: Term 1 election: Requested vote from peers b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465), c04263dc1b6d4b88b073d5d0ddfb3a35 (127.11.123.129:38089)
I20260620 07:19:57.773412 27826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64bc93764804346a7c7729f9c3da4bc" candidate_uuid: "bf7520db1f0548958c30ac156f6a3b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35"
I20260620 07:19:57.773425 27964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64bc93764804346a7c7729f9c3da4bc" candidate_uuid: "bf7520db1f0548958c30ac156f6a3b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b265150651a54d5f9d62064de91ddc26"
I20260620 07:19:57.773541 27826 raft_consensus.cc:3037] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:57.773527 27964 raft_consensus.cc:3037] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:19:57.774525 27964 raft_consensus.cc:2454] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7520db1f0548958c30ac156f6a3b71 in term 1.
I20260620 07:19:57.774605 27826 raft_consensus.cc:2454] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7520db1f0548958c30ac156f6a3b71 in term 1.
I20260620 07:19:57.774843 28029 leader_election.cc:302] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b265150651a54d5f9d62064de91ddc26, bf7520db1f0548958c30ac156f6a3b71; no voters:
I20260620 07:19:57.775018 28160 raft_consensus.cc:2781] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:19:57.775213 28160 raft_consensus.cc:686] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 1 LEADER]: Becoming Leader. State: Replica: bf7520db1f0548958c30ac156f6a3b71, State: Running, Role: LEADER
I20260620 07:19:57.775532 28160 consensus_queue.cc:237] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:19:57.778607 27684 catalog_manager.cc:5495] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131). New cstate: current_term: 1 leader_uuid: "bf7520db1f0548958c30ac156f6a3b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20260620 07:19:57.803004 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.799898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.11.123.131:37943)
0620 07:19:57.800011 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 07:19:57.800017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:19:57.800062 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:19:57.800186 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:19:57.800188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:19:57.800237 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:19:57.800319 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:19:57.800515 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.800980 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.801806 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:19:57.802025 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:19:57.802110 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:19:57.802369 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:19:57.802730 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:19:57.802822 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:19:57.802872 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 07:19:57.802917 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:19:57.809052 27826 raft_consensus.cc:1261] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Refusing update from remote peer bf7520db1f0548958c30ac156f6a3b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:19:57.809137 27964 raft_consensus.cc:1261] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26 [term 1 FOLLOWER]: Refusing update from remote peer bf7520db1f0548958c30ac156f6a3b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:19:57.809790 28160 consensus_queue.cc:1035] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:19:57.810029 28163 consensus_queue.cc:1035] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:19:57.815795 28165 mvcc.cc:205] Tried to move back new op lower bound from 7298826231018876928 to 7298826230891159552. Current Snapshot: MvccSnapshot[applied={T|T < 7298826231018876928}]
I20260620 07:19:57.824432 27684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:19:57.824590 27684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:19:57.826682 27684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Sending DeleteTablet for 3 replicas of tablet f64bc93764804346a7c7729f9c3da4bc
I20260620 07:19:57.827316 28076 tablet_service.cc:1514] Processing DeleteTablet for tablet f64bc93764804346a7c7729f9c3da4bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:57 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:38864
I20260620 07:19:57.827452 27944 tablet_service.cc:1514] Processing DeleteTablet for tablet f64bc93764804346a7c7729f9c3da4bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:57 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:54822
I20260620 07:19:57.827520 27806 tablet_service.cc:1514] Processing DeleteTablet for tablet f64bc93764804346a7c7729f9c3da4bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:19:57 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:55302
I20260620 07:19:57.827872 28176 tablet_replica.cc:330] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: stopping tablet replica
I20260620 07:19:57.827957 28174 tablet_replica.cc:330] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71: stopping tablet replica
I20260620 07:19:57.828047 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 27722
I20260620 07:19:57.828101 28175 tablet_replica.cc:330] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: stopping tablet replica
I20260620 07:19:57.828299 28174 raft_consensus.cc:2229] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:19:57.828799 28174 raft_consensus.cc:2258] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:19:57.833878 28174 ts_tablet_manager.cc:1905] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:19:57.834254 28175 raft_consensus.cc:2229] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:19:57.834605 28175 raft_consensus.cc:2258] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 07:19:57.835569 27668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:38089 (error 108)
I20260620 07:19:57.836054 28175 ts_tablet_manager.cc:1905] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:19:57.836781 28174 ts_tablet_manager.cc:1918] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:19:57.836941 28174 log.cc:1196] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/f64bc93764804346a7c7729f9c3da4bc
I20260620 07:19:57.837303 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.836955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:38089 (local address 127.0.0.1:55304)
0620 07:19:57.837147 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:57.837227 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 07:19:57.837486 28174 ts_tablet_manager.cc:1939] T f64bc93764804346a7c7729f9c3da4bc P bf7520db1f0548958c30ac156f6a3b71: Deleting consensus metadata
W20260620 07:19:57.837512 27668 catalog_manager.cc:4584] TS c04263dc1b6d4b88b073d5d0ddfb3a35 (127.11.123.129:38089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f64bc93764804346a7c7729f9c3da4bc: Network error: Client connection negotiation failed: client connection to 127.11.123.129:38089: connect: Connection refused (error 111)
I20260620 07:19:57.838369 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 27876
I20260620 07:19:57.840835 27671 catalog_manager.cc:4841] TS bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943): tablet f64bc93764804346a7c7729f9c3da4bc (table test-table [id=ef7b029546aa4716bd9ccf948a199f8d]) successfully deleted
I20260620 07:19:57.845295 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 28013
I20260620 07:19:57.847040 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:19:57.845487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:39465 (local address 127.0.0.1:54830)
0620 07:19:57.846902 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0620 07:19:57.846966 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.130:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1317}
I20260620 07:19:57.850049 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 27651
I20260620 07:20:17.856506 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:36513
--webserver_interface=127.11.123.190
--webserver_port=35253
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:17.970175 28178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:17.970460 28178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:17.970546 28178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:17.974534 28178 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:20:17.974601 28178 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:20:17.974627 28178 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:20:17.974658 28178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:17.974709 28178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:17.974766 28178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:17.974804 28178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:17.974845 28178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:17.979931 28178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:17.980926 28178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:17.981945 28178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:17.990573 28184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:17.990674 28178 server_base.cc:1034] running on GCE node
W20260620 07:20:17.990594 28183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:17.990626 28186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:17.991359 28178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:17.992393 28178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:17.993579 28178 hybrid_clock.cc:648] HybridClock initialized: now 1781940017993557 us; error 47 us; skew 500 ppm
Jun 20 07:20:17 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940017, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:17.996203 28178 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:20:17.997035 28178 webserver.cc:460] Webserver started at http://127.11.123.190:35253/ using document root <none> and password file <none>
I20260620 07:20:17.997586 28178 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:17.997679 28178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:18.001415 28178 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 07:20:18.003418 28193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:18.004627 28178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:20:18.004765 28178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "70181d3e668a4047a20e9ce06ba3333c"
format_stamp: "Formatted at 2026-06-20 07:19:35 on dist-test-slave-j663"
server_key: "fcaded7a7120329ffe174b64ac0a7416"
server_key_iv: "d41577a7804a45959f02142afd826b78"
server_key_version: "encryptionkey@0"
I20260620 07:20:18.005345 28178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:18.014662 28178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:18.017534 28178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:18.017725 28178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:18.025303 28178 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:36513
I20260620 07:20:18.025619 28245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:36513 every 8 connection(s)
I20260620 07:20:18.026396 28178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:18.030682 28246 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:20:18.032284 28246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Bootstrap starting.
I20260620 07:20:18.034117 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 28178
I20260620 07:20:18.034564 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:38089
--local_ip_for_outbound_sockets=127.11.123.129
--tserver_master_addrs=127.11.123.190:36513
--webserver_port=45485
--webserver_interface=127.11.123.129
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:20:18.049506 28246 log.cc:824] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:18.060662 28246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:20:18.061041 28246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Bootstrap complete.
I20260620 07:20:18.064904 28246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } }
I20260620 07:20:18.065346 28246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70181d3e668a4047a20e9ce06ba3333c, State: Initialized, Role: FOLLOWER
I20260620 07:20:18.065819 28246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } }
I20260620 07:20:18.065927 28246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:18.066015 28246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:18.066133 28246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 2 FOLLOWER]: Advancing to term 3
I20260620 07:20:18.067917 28246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } }
I20260620 07:20:18.068240 28246 leader_election.cc:302] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70181d3e668a4047a20e9ce06ba3333c; no voters:
I20260620 07:20:18.068559 28246 leader_election.cc:288] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 07:20:18.068647 28250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 3 FOLLOWER]: Leader election won for term 3
I20260620 07:20:18.068838 28250 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [term 3 LEADER]: Becoming Leader. State: Replica: 70181d3e668a4047a20e9ce06ba3333c, State: Running, Role: LEADER
I20260620 07:20:18.069205 28250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } }
I20260620 07:20:18.069765 28246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:18.070087 28250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70181d3e668a4047a20e9ce06ba3333c. Latest consensus state: current_term: 3 leader_uuid: "70181d3e668a4047a20e9ce06ba3333c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } } }
I20260620 07:20:18.070204 28250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:18.070574 28251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "70181d3e668a4047a20e9ce06ba3333c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70181d3e668a4047a20e9ce06ba3333c" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 36513 } } }
I20260620 07:20:18.070629 28254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:18.071940 28251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:18.075593 28254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:18.076386 28254 catalog_manager.cc:1259] Loaded cluster ID: c5cdf6d76de44e19acd2415f7cbe8e82
I20260620 07:20:18.076444 28254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:18.078830 28254 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:18.079631 28254 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Loaded TSK: 0
I20260620 07:20:18.079732 28254 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 70181d3e668a4047a20e9ce06ba3333c: Loaded TSK: 1
F20260620 07:20:18.080616 28254 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb6fa7ce9ad google::LogMessage::Fail() at ??:0
@ 0x7fb6fa7d188a google::LogMessage::SendToLog() at ??:0
@ 0x7fb6fa7ce4de google::LogMessage::Flush() at ??:0
@ 0x7fb6fa7cfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb6fd49218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb6fd48af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb7051337bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb70510d99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb70510e178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fb70514fdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb70516e82b std::function<>::operator()() at ??:0
@ 0x7fb70510db4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fb70510ef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb70510b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fb70514f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb6fca203d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb6fca20c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb6fca223f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb6fca11512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb6fe0556db start_thread at ??:0
@ 0x7fb6f92cc61f clone at ??:0
W20260620 07:20:18.172817 28248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:18.173034 28248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:18.173141 28248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:18.177495 28248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:18.177538 28248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:18.177597 28248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:20:18.183487 28248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:38089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=45485
--tserver_master_addrs=127.11.123.190:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:18.184479 28248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:18.185459 28248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:18.195060 28294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:18.195070 28293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:18.195075 28296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:18.195289 28248 server_base.cc:1034] running on GCE node
I20260620 07:20:18.195820 28248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:18.196466 28248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:18.197650 28248 hybrid_clock.cc:648] HybridClock initialized: now 1781940018197618 us; error 49 us; skew 500 ppm
Jun 20 07:20:18 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940018, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:18.200436 28248 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:20:18.201479 28248 webserver.cc:460] Webserver started at http://127.11.123.129:45485/ using document root <none> and password file <none>
I20260620 07:20:18.202024 28248 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:18.202116 28248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:18.205940 28248 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:20:18.208056 28303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:18.209412 28248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:20:18.209532 28248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35"
format_stamp: "Formatted at 2026-06-20 07:19:35 on dist-test-slave-j663"
server_key: "18d008f87e5a1963630f4d0bcb08d319"
server_key_iv: "019beb13dd882df2b67eaad31eba1dec"
server_key_version: "encryptionkey@0"
I20260620 07:20:18.210048 28248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:18.225791 28248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:18.228911 28248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:18.229139 28248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:18.229817 28248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:18.230991 28310 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 07:20:18.236181 28248 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:20:18.236261 28248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 07:20:18.236339 28248 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:20:18.238013 28248 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:20:18.238080 28248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 07:20:18.238363 28310 tablet_bootstrap.cc:492] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: Bootstrap starting.
I20260620 07:20:18.248601 28248 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:38089
I20260620 07:20:18.249653 28248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:20:18.251196 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 28248
I20260620 07:20:18.251715 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:39465
--local_ip_for_outbound_sockets=127.11.123.130
--tserver_master_addrs=127.11.123.190:36513
--webserver_port=38031
--webserver_interface=127.11.123.130
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:20:18.253415 28417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:38089 every 8 connection(s)
I20260620 07:20:18.260907 28310 log.cc:824] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:18.270385 28310 tablet_bootstrap.cc:492] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:20:18.270787 28310 tablet_bootstrap.cc:492] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: Bootstrap complete.
I20260620 07:20:18.272176 28310 ts_tablet_manager.cc:1397] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: Time spent bootstrapping tablet: real 0.034s user 0.030s sys 0.000s
I20260620 07:20:18.275358 28310 raft_consensus.cc:348] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:20:18.275879 28310 raft_consensus.cc:729] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04263dc1b6d4b88b073d5d0ddfb3a35, State: Initialized, Role: FOLLOWER
I20260620 07:20:18.276382 28310 consensus_queue.cc:260] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:20:18.277386 28310 ts_tablet_manager.cc:1428] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260620 07:20:18.325858 28420 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:20:18.251716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:57777)
0620 07:20:18.252396 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:18.252476 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:18.253700 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:18.325004 (+ 71304us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":276,"threads_started":1}
W20260620 07:20:18.326378 28418 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:36513 (0 consecutive failures): Network error: Failed to ping master at 127.11.123.190:36513: Client connection negotiation failed: client connection to 127.11.123.190:36513: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 07:20:18.327111 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:18.326848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:43693)
0620 07:20:18.326979 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:18.327056 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 07:20:18.327749 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:18.327540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:39213)
0620 07:20:18.327648 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:18.327693 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:20:18.327910 28418 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:20:18.372367 28421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:18.372687 28421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:18.372756 28421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:18.379724 28421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:18.379788 28421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:18.379848 28421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:20:18.385002 28421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:39465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=38031
--tserver_master_addrs=127.11.123.190:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:18.385995 28421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:18.386950 28421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:18.396309 28429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:18.396330 28431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:18.396293 28428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:18.396512 28421 server_base.cc:1034] running on GCE node
I20260620 07:20:18.396947 28421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:18.397616 28421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:18.398777 28421 hybrid_clock.cc:648] HybridClock initialized: now 1781940018398767 us; error 29 us; skew 500 ppm
Jun 20 07:20:18 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940018, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:18.401571 28421 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:20:18.402375 28421 webserver.cc:460] Webserver started at http://127.11.123.130:38031/ using document root <none> and password file <none>
I20260620 07:20:18.402868 28421 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:18.402926 28421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:18.406577 28421 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:20:18.408752 28438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:18.410043 28421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:20:18.410164 28421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "b265150651a54d5f9d62064de91ddc26"
format_stamp: "Formatted at 2026-06-20 07:19:36 on dist-test-slave-j663"
server_key: "29711b8b415aa407bc513f60a4c397cf"
server_key_iv: "ae7c63287caba4ffde5fe2c944cffb17"
server_key_version: "encryptionkey@0"
I20260620 07:20:18.410691 28421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:18.426345 28421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:18.429533 28421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:18.429728 28421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:18.430297 28421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:18.431314 28445 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 07:20:18.435223 28445 ts_tablet_manager.cc:1852] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 07:20:18.435611 28445 ts_tablet_manager.cc:1905] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:20:18.439429 28445 ts_tablet_manager.cc:1918] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:20:18.439522 28445 log.cc:1196] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/f64bc93764804346a7c7729f9c3da4bc
I20260620 07:20:18.439883 28445 ts_tablet_manager.cc:1939] T f64bc93764804346a7c7729f9c3da4bc P b265150651a54d5f9d62064de91ddc26: Deleting consensus metadata
I20260620 07:20:18.440203 28421 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 07:20:18.440279 28421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260620 07:20:18.440379 28421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:18.440421 28421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:18.450109 28421 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:39465
I20260620 07:20:18.450150 28552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:39465 every 8 connection(s)
I20260620 07:20:18.450941 28421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:20:18.452080 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 28421
I20260620 07:20:18.452509 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:37943
--local_ip_for_outbound_sockets=127.11.123.131
--tserver_master_addrs=127.11.123.190:36513
--webserver_port=45739
--webserver_interface=127.11.123.131
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:20:18.454505 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:18.453176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:43697)
0620 07:20:18.453653 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:18.453779 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":135,"threads_started":1}
W20260620 07:20:18.455037 28553 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:36513 (0 consecutive failures): Network error: Failed to ping master at 127.11.123.190:36513: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
I20260620 07:20:18.455606 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:18.455386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:46747)
0620 07:20:18.455499 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:18.455548 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 07:20:18.456368 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:18.456118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:58365)
0620 07:20:18.456256 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:18.456307 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:20:18.456589 28553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:20:18.561820 28556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:18.562055 28556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:18.562137 28556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:18.566051 28556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:18.566103 28556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:18.566206 28556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:20:18.571383 28556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:37943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=45739
--tserver_master_addrs=127.11.123.190:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:18.572337 28556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:18.573397 28556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:18.582636 28561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:18.582651 28564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:18.582644 28562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:18.582981 28556 server_base.cc:1034] running on GCE node
I20260620 07:20:18.583362 28556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:18.584002 28556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:18.585150 28556 hybrid_clock.cc:648] HybridClock initialized: now 1781940018585114 us; error 50 us; skew 500 ppm
Jun 20 07:20:18 dist-test-slave-j663 krb5kdc[27104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940018, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:18.587662 28556 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:20:18.588455 28556 webserver.cc:460] Webserver started at http://127.11.123.131:45739/ using document root <none> and password file <none>
I20260620 07:20:18.589003 28556 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:18.589109 28556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:18.592727 28556 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:20:18.594748 28571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:18.595880 28556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:20:18.596043 28556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "bf7520db1f0548958c30ac156f6a3b71"
format_stamp: "Formatted at 2026-06-20 07:19:36 on dist-test-slave-j663"
server_key: "e09c55822ab3d6fb6674223e5a5e27a4"
server_key_iv: "287309880c95c36037e008ee0279a687"
server_key_version: "encryptionkey@0"
I20260620 07:20:18.596580 28556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:18.616703 28556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:18.619896 28556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:18.620102 28556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:18.620671 28556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:18.621512 28556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:18.621590 28556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:18.621659 28556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:18.621702 28556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:18.630538 28556 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:37943
I20260620 07:20:18.630570 28684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:37943 every 8 connection(s)
I20260620 07:20:18.631469 28556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:20:18.634909 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:18.633394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:38103)
0620 07:20:18.633845 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:18.634253 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
W20260620 07:20:18.635380 28685 heartbeater.cc:644] Failed to heartbeat to 127.11.123.190:36513 (0 consecutive failures): Network error: Failed to ping master at 127.11.123.190:36513: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
I20260620 07:20:18.635949 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:18.635753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:35887)
0620 07:20:18.635854 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:18.635899 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 07:20:18.636559 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:18.636363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:50733)
0620 07:20:18.636459 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:18.636513 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:20:18.636730 28685 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:20:18.638938 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 28556
I20260620 07:20:19.328890 28689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:19.328452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:43435)
0620 07:20:19.328739 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:19.328790 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":120,"threads_started":1}
I20260620 07:20:19.457626 28690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:19.457259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:46975)
0620 07:20:19.457491 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:19.457543 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":64,"threads_started":1}
I20260620 07:20:19.562342 28691 raft_consensus.cc:482] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:20:19.562525 28691 raft_consensus.cc:504] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:20:19.563561 28691 leader_election.cc:288] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943), b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465)
Jun 20 07:20:19 dist-test-slave-j663 krb5kdc[27104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940018, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.131@KRBTEST.COM
Jun 20 07:20:19 dist-test-slave-j663 krb5kdc[27104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940018, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.130@KRBTEST.COM
I20260620 07:20:19.578197 28692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:19.563798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:39465 (local address 127.11.123.129:49075)
0620 07:20:19.564200 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:19.564215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:19.564313 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:19.566563 (+ 2250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:19.566566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:19.566728 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:19.566978 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:19.566992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:19.568410 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:19.568413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:19.569400 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:19.569410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:19.570110 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:19.570114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:19.570295 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:19.571009 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:20:19.571020 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:19.572913 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:19.575195 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:19.575198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:19.575200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:19.575666 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:19.576074 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:19.576077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:19.576079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:19.576264 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:19.577022 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:19.577028 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:19.577379 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 07:20:19.577595 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 07:20:19.577982 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":183,"spinlock_wait_cycles":29696,"thread_start_us":71,"threads_started":1}
I20260620 07:20:19.578234 28693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:19.563923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:48115 (local address 127.11.123.131:37943)
0620 07:20:19.564318 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:19.564334 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:19.564376 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:20:19.565291 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:19.565300 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:19.565754 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:19.566085 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:19.567170 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:19.568176 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:19.569518 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:19.569973 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:19.570147 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:19.572804 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:19.572809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:19.572821 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:19.572849 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:19.575013 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:19.575791 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:19.575797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:19.575803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:19.575967 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:19.576367 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:19.576371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:19.576373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:19.576887 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:19.577034 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:19.577769 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 07:20:19.578055 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":87,"threads_started":1}
I20260620 07:20:19.578240 28694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:19.563975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:49075 (local address 127.11.123.130:39465)
0620 07:20:19.564318 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:19.564334 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:19.564376 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:20:19.565313 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:19.565319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:19.565753 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:19.566085 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:19.567185 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:19.568230 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:19.569531 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:19.569983 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:19.570163 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:19.573149 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:19.573156 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:19.573173 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:19.573213 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:19.575098 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:19.575791 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:19.575797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:19.575803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:19.575967 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:19.576367 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:19.576371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:19.576373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:19.576887 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:19.577034 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:19.577790 (+ 756us) server_negotiation.cc:299] Negotiation successful
0620 07:20:19.578055 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260620 07:20:19.578197 28689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:19.563755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:37943 (local address 127.11.123.129:48115)
0620 07:20:19.563937 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:19.563952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:19.564064 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:19.566160 (+ 2096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:19.566164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:19.566529 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:19.566978 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:19.566991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:19.568325 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:19.568330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:19.569394 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:19.569403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:19.570101 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:19.570104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:19.570298 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:19.570949 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:20:19.570965 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:19.572561 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:19.575132 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:19.575136 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:19.575169 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:19.575686 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:19.576074 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:19.576077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:19.576079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:19.576264 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:19.577022 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:19.577028 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:19.577379 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 07:20:19.577595 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 07:20:19.578034 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":234,"spinlock_wait_cycles":54272}
W20260620 07:20:19.580892 28304 leader_election.cc:341] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465): Not found: Tablet not found: f64bc93764804346a7c7729f9c3da4bc
W20260620 07:20:19.581488 28307 leader_election.cc:341] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943): Not found: Tablet not found: f64bc93764804346a7c7729f9c3da4bc
I20260620 07:20:19.581579 28307 leader_election.cc:302] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04263dc1b6d4b88b073d5d0ddfb3a35; no voters: b265150651a54d5f9d62064de91ddc26, bf7520db1f0548958c30ac156f6a3b71
I20260620 07:20:19.581858 28691 raft_consensus.cc:2726] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:20:19.637681 28695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:19.637287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:37071)
0620 07:20:19.637540 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:19.637593 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260620 07:20:20.330083 28696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:20.329643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:44889)
0620 07:20:20.329918 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:20.329979 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260620 07:20:20.458714 28697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:20.458325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:54579)
0620 07:20:20.458567 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:20.458618 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":95,"threads_started":1}
I20260620 07:20:20.638818 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:20.638380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:54027)
0620 07:20:20.638652 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:20.638710 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
I20260620 07:20:21.331311 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:21.330901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:39861)
0620 07:20:21.331164 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:21.331217 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260620 07:20:21.459882 28700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:21.459441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:47899)
0620 07:20:21.459738 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:21.459788 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":89,"threads_started":1}
I20260620 07:20:21.542685 28701 raft_consensus.cc:482] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:20:21.542819 28701 raft_consensus.cc:504] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:20:21.543219 28701 leader_election.cc:288] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943), b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465)
W20260620 07:20:21.544169 28307 leader_election.cc:341] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943): Not found: Tablet not found: f64bc93764804346a7c7729f9c3da4bc
W20260620 07:20:21.544284 28304 leader_election.cc:341] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465): Not found: Tablet not found: f64bc93764804346a7c7729f9c3da4bc
I20260620 07:20:21.544349 28304 leader_election.cc:302] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04263dc1b6d4b88b073d5d0ddfb3a35; no voters: b265150651a54d5f9d62064de91ddc26, bf7520db1f0548958c30ac156f6a3b71
I20260620 07:20:21.544543 28701 raft_consensus.cc:2726] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:20:21.640298 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:21.639686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:51529)
0620 07:20:21.640095 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:21.640165 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":114,"threads_started":1}
I20260620 07:20:22.332417 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:22.332043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:48925)
0620 07:20:22.332272 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:22.332323 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260620 07:20:22.461016 28704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:22.460598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:37863)
0620 07:20:22.460847 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:22.460909 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260620 07:20:22.641505 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:22.641064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:37569)
0620 07:20:22.641366 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:22.641420 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":82,"threads_started":1}
I20260620 07:20:23.333765 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:23.333359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:42375)
0620 07:20:23.333625 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:23.333673 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":68,"threads_started":1}
I20260620 07:20:23.462266 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:23.461885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:42313)
0620 07:20:23.462129 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:23.462181 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
I20260620 07:20:23.642680 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:23.642280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:32895)
0620 07:20:23.642519 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:23.642586 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260620 07:20:24.334854 28709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:24.334440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:58323)
0620 07:20:24.334705 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:24.334760 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260620 07:20:24.391207 28710 raft_consensus.cc:482] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:20:24.391356 28710 raft_consensus.cc:504] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:20:24.391793 28710 leader_election.cc:288] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943), b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465)
W20260620 07:20:24.392598 28307 leader_election.cc:341] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943): Not found: Tablet not found: f64bc93764804346a7c7729f9c3da4bc
W20260620 07:20:24.392699 28304 leader_election.cc:341] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465): Not found: Tablet not found: f64bc93764804346a7c7729f9c3da4bc
I20260620 07:20:24.392761 28304 leader_election.cc:302] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04263dc1b6d4b88b073d5d0ddfb3a35; no voters: b265150651a54d5f9d62064de91ddc26, bf7520db1f0548958c30ac156f6a3b71
I20260620 07:20:24.392894 28710 raft_consensus.cc:2726] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:20:24.463439 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:24.462992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:35057)
0620 07:20:24.463252 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:24.463304 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260620 07:20:24.643872 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:24.643462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:52743)
0620 07:20:24.643729 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:24.643781 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":95,"threads_started":1}
I20260620 07:20:25.336024 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:25.335629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:45005)
0620 07:20:25.335875 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:25.335930 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260620 07:20:25.464516 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:25.464080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:45661)
0620 07:20:25.464373 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:25.464427 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":69,"threads_started":1}
I20260620 07:20:25.645040 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:25.644596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:44015)
0620 07:20:25.644894 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:25.644949 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":71,"threads_started":1}
I20260620 07:20:26.239894 28716 raft_consensus.cc:482] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:20:26.240021 28716 raft_consensus.cc:504] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:20:26.240478 28716 leader_election.cc:288] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943), b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465)
W20260620 07:20:26.241389 28307 leader_election.cc:341] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943): Not found: Tablet not found: f64bc93764804346a7c7729f9c3da4bc
W20260620 07:20:26.241501 28304 leader_election.cc:341] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465): Not found: Tablet not found: f64bc93764804346a7c7729f9c3da4bc
I20260620 07:20:26.241595 28304 leader_election.cc:302] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04263dc1b6d4b88b073d5d0ddfb3a35; no voters: b265150651a54d5f9d62064de91ddc26, bf7520db1f0548958c30ac156f6a3b71
I20260620 07:20:26.241752 28716 raft_consensus.cc:2726] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:20:26.337270 28717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:26.336779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:44665)
0620 07:20:26.337062 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:26.337172 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":123,"threads_started":1}
I20260620 07:20:26.465801 28718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:26.465332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:59319)
0620 07:20:26.465641 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:26.465706 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":148,"threads_started":1}
I20260620 07:20:26.646267 28719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:26.645853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:33761)
0620 07:20:26.646116 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:26.646171 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":105,"threads_started":1}
I20260620 07:20:27.338408 28720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:27.338024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:44459)
0620 07:20:27.338264 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:27.338316 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260620 07:20:27.466953 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:27.466558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:46113)
0620 07:20:27.466809 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:27.466864 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260620 07:20:27.647554 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:27.647032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:35827)
0620 07:20:27.647399 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:27.647457 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
I20260620 07:20:28.339668 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:28.339253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:55159)
0620 07:20:28.339511 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:28.339574 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":86,"threads_started":1}
I20260620 07:20:28.468134 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:28.467729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:49901)
0620 07:20:28.467979 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:28.468034 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260620 07:20:28.648803 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:28.648332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:38259)
0620 07:20:28.648652 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:28.648707 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":132,"threads_started":1}
I20260620 07:20:29.340945 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:29.340422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:40317)
0620 07:20:29.340755 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:29.340814 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":134,"threads_started":1}
I20260620 07:20:29.469336 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:29.468839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:59355)
0620 07:20:29.469133 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:29.469190 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260620 07:20:29.649971 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:29.649551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:43669)
0620 07:20:29.649821 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:29.649880 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
I20260620 07:20:30.308985 28729 raft_consensus.cc:482] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:20:30.309218 28729 raft_consensus.cc:504] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7520db1f0548958c30ac156f6a3b71" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 37943 } } peers { permanent_uuid: "b265150651a54d5f9d62064de91ddc26" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 39465 } } peers { permanent_uuid: "c04263dc1b6d4b88b073d5d0ddfb3a35" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 38089 } }
I20260620 07:20:30.309679 28729 leader_election.cc:288] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943), b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465)
W20260620 07:20:30.310391 28307 leader_election.cc:341] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf7520db1f0548958c30ac156f6a3b71 (127.11.123.131:37943): Not found: Tablet not found: f64bc93764804346a7c7729f9c3da4bc
W20260620 07:20:30.310511 28304 leader_election.cc:341] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b265150651a54d5f9d62064de91ddc26 (127.11.123.130:39465): Not found: Tablet not found: f64bc93764804346a7c7729f9c3da4bc
I20260620 07:20:30.310634 28304 leader_election.cc:302] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04263dc1b6d4b88b073d5d0ddfb3a35; no voters: b265150651a54d5f9d62064de91ddc26, bf7520db1f0548958c30ac156f6a3b71
I20260620 07:20:30.310778 28729 raft_consensus.cc:2726] T f64bc93764804346a7c7729f9c3da4bc P c04263dc1b6d4b88b073d5d0ddfb3a35 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:20:30.342043 28730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:30.341657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:52683)
0620 07:20:30.341899 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:30.341952 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260620 07:20:30.470472 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:30.470053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:47257)
0620 07:20:30.470328 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:30.470380 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
I20260620 07:20:30.651153 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:30.650766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:56165)
0620 07:20:30.651010 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:30.651064 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":69,"threads_started":1}
I20260620 07:20:31.343329 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:31.342897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:48013)
0620 07:20:31.343177 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:31.343236 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260620 07:20:31.471654 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:31.471199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:46615)
0620 07:20:31.471494 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:31.471553 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
I20260620 07:20:31.652329 28735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:31.651930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:42373)
0620 07:20:31.652182 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:31.652236 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260620 07:20:32.344785 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:32.344197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:51857)
0620 07:20:32.344587 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:32.344662 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":96,"threads_started":1}
I20260620 07:20:32.472900 28737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:32.472493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:39603)
0620 07:20:32.472750 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:32.472808 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
I20260620 07:20:32.653524 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:32.653028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.131:46955)
0620 07:20:32.653329 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:32.653385 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":143,"threads_started":1}
I20260620 07:20:33.346240 28739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:33.345689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.129:44257)
0620 07:20:33.346034 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:33.346102 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":98,"threads_started":1}
I20260620 07:20:33.474134 28740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:33.473753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:36513 (local address 127.11.123.130:57481)
0620 07:20:33.473983 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:33.474032 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.190:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260620 07:20:33.640002 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 28248
I20260620 07:20:33.646136 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 28421
I20260620 07:20:33.651764 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 28556
2026-06-20T07:20:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59396 ms)
[----------] 30 tests from SecurityITest (239621 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:33 dist-test-slave-j663 krb5kdc[28743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:33 dist-test-slave-j663 krb5kdc[28743](info): set up 2 sockets
Jun 20 07:20:33 dist-test-slave-j663 krb5kdc[28743](info): commencing operation
krb5kdc: starting...
W20260620 07:20:34.713217 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:34 dist-test-slave-j663 krb5kdc[28743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:20:34.844676 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:46063
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:46063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:34.961480 28756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:34.961771 28756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:34.961814 28756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:34.965950 28756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:34.966013 28756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:34.966032 28756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:34.966050 28756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:34.966066 28756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:34.971551 28756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:46063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:46063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:34.972594 28756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:34.973641 28756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:34.982287 28764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:34.982287 28761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:34.982372 28762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:34.982470 28756 server_base.cc:1034] running on GCE node
I20260620 07:20:34.983271 28756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:34.984349 28756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:34.985548 28756 hybrid_clock.cc:648] HybridClock initialized: now 1781940034985512 us; error 49 us; skew 500 ppm
Jun 20 07:20:34 dist-test-slave-j663 krb5kdc[28743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940034, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:34.989219 28756 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:20:34.990311 28756 webserver.cc:460] Webserver started at http://127.11.123.190:44199/ using document root <none> and password file <none>
I20260620 07:20:34.991017 28756 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:34.991140 28756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:34.991405 28756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:34.994094 28756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "2a311df401554fe58638ece739540a14"
format_stamp: "Formatted at 2026-06-20 07:20:34 on dist-test-slave-j663"
server_key: "f2e45068d1d1e9829a27a9c11b5a99de"
server_key_iv: "f1cc180141e1b9f73c6e94913e0c85e8"
server_key_version: "encryptionkey@0"
I20260620 07:20:34.994789 28756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "2a311df401554fe58638ece739540a14"
format_stamp: "Formatted at 2026-06-20 07:20:34 on dist-test-slave-j663"
server_key: "f2e45068d1d1e9829a27a9c11b5a99de"
server_key_iv: "f1cc180141e1b9f73c6e94913e0c85e8"
server_key_version: "encryptionkey@0"
I20260620 07:20:34.998801 28756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 07:20:35.001518 28771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:35.003038 28756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:20:35.003196 28756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "2a311df401554fe58638ece739540a14"
format_stamp: "Formatted at 2026-06-20 07:20:34 on dist-test-slave-j663"
server_key: "f2e45068d1d1e9829a27a9c11b5a99de"
server_key_iv: "f1cc180141e1b9f73c6e94913e0c85e8"
server_key_version: "encryptionkey@0"
I20260620 07:20:35.003322 28756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:35.019260 28756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:35.023326 28756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:35.023546 28756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:35.030822 28756 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:46063
I20260620 07:20:35.030846 28823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:46063 every 8 connection(s)
I20260620 07:20:35.031848 28756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:35.034564 28824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:35.039884 28824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14: Bootstrap starting.
I20260620 07:20:35.041646 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 28756
I20260620 07:20:35.041778 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:35.042079 11758 external_mini_cluster.cc:1428] Setting key d8ce7a42fbfbc3a8b00d83eb3170b3f4
I20260620 07:20:35.042384 28824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:35.043188 28824 log.cc:824] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:35.045236 28824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14: No bootstrap required, opened a new log
I20260620 07:20:35.048700 28824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a311df401554fe58638ece739540a14" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46063 } }
I20260620 07:20:35.048975 28824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:35.049100 28824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a311df401554fe58638ece739540a14, State: Initialized, Role: FOLLOWER
I20260620 07:20:35.049666 28824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a311df401554fe58638ece739540a14" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46063 } }
I20260620 07:20:35.049794 28824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:35.049873 28824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:35.049973 28824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 07:20:35 dist-test-slave-j663 krb5kdc[28743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:20:35.051142 28824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a311df401554fe58638ece739540a14" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46063 } }
I20260620 07:20:35.051474 28824 leader_election.cc:302] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a311df401554fe58638ece739540a14; no voters:
I20260620 07:20:35.051834 28824 leader_election.cc:288] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:35.051928 28829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:35.052107 28829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [term 1 LEADER]: Becoming Leader. State: Replica: 2a311df401554fe58638ece739540a14, State: Running, Role: LEADER
I20260620 07:20:35.052384 28829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a311df401554fe58638ece739540a14" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46063 } }
I20260620 07:20:35.053150 28824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:35.053591 28831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a311df401554fe58638ece739540a14. Latest consensus state: current_term: 1 leader_uuid: "2a311df401554fe58638ece739540a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a311df401554fe58638ece739540a14" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46063 } } }
I20260620 07:20:35.053704 28831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:35.054003 28830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a311df401554fe58638ece739540a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a311df401554fe58638ece739540a14" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 46063 } } }
I20260620 07:20:35.054229 28830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:35.054744 28833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:35.057482 28827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.043196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.11.123.190:46063)
0620 07:20:35.043671 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.043693 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.043738 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:20:35.044432 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.044451 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.044779 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.045157 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:35.045930 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.046894 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.047874 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.048345 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.048529 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.051637 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:35.051642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:35.051655 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:35.051682 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:35.054019 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:35.054575 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:35.054581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:35.054587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:35.054765 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:35.055115 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:35.055119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:35.055120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:35.055660 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:35.055885 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.056148 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.056480 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":137,"threads_started":1}
I20260620 07:20:35.058195 28833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:35.064208 28833 catalog_manager.cc:1347] Generated new cluster ID: e6e31dda30ae4a49a8d3cf961397939e
I20260620 07:20:35.064291 28833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:35.081044 28833 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:35.082350 28833 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:35.103811 28833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14: Generated new TSK 0
I20260620 07:20:35.104712 28833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:20:35.146165 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46063
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:35.264230 28850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:35.264535 28850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:35.264619 28850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:35.268841 28850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:35.268949 28850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:35.269045 28850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:20:35.274396 28850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:35.275664 28850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:35.276751 28850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:35.287384 28856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:35.287395 28858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:35.287412 28855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:35.288121 28850 server_base.cc:1034] running on GCE node
I20260620 07:20:35.288622 28850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:35.289546 28850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:35.290738 28850 hybrid_clock.cc:648] HybridClock initialized: now 1781940035290725 us; error 36 us; skew 500 ppm
Jun 20 07:20:35 dist-test-slave-j663 krb5kdc[28743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940035, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:35.294425 28850 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:20:35.295423 28850 webserver.cc:460] Webserver started at http://127.11.123.129:46025/ using document root <none> and password file <none>
I20260620 07:20:35.296065 28850 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:35.296139 28850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:35.296337 28850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:35.298954 28850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "0ba086b4149040b497a2b006a82233d8"
format_stamp: "Formatted at 2026-06-20 07:20:35 on dist-test-slave-j663"
server_key: "81143fd573c4ae64484f981ac91b01d9"
server_key_iv: "a8bcaf7e5c7106c591bc116759c13104"
server_key_version: "encryptionkey@0"
I20260620 07:20:35.299599 28850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "0ba086b4149040b497a2b006a82233d8"
format_stamp: "Formatted at 2026-06-20 07:20:35 on dist-test-slave-j663"
server_key: "81143fd573c4ae64484f981ac91b01d9"
server_key_iv: "a8bcaf7e5c7106c591bc116759c13104"
server_key_version: "encryptionkey@0"
I20260620 07:20:35.303558 28850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:20:35.306021 28865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:35.307390 28850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:20:35.307513 28850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "0ba086b4149040b497a2b006a82233d8"
format_stamp: "Formatted at 2026-06-20 07:20:35 on dist-test-slave-j663"
server_key: "81143fd573c4ae64484f981ac91b01d9"
server_key_iv: "a8bcaf7e5c7106c591bc116759c13104"
server_key_version: "encryptionkey@0"
I20260620 07:20:35.307657 28850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:35.319859 28850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:35.323293 28850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:35.323511 28850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:35.324158 28850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:35.325048 28850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:35.325173 28850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:35.325250 28850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:35.325282 28850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:35.334704 28850 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:41059
I20260620 07:20:35.334755 28978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:41059 every 8 connection(s)
I20260620 07:20:35.335656 28850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:20:35.343395 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 28850
I20260620 07:20:35.343544 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:20:35.343876 11758 external_mini_cluster.cc:1428] Setting key ab3e15ff59ee844e6265b230e3312bf3
Jun 20 07:20:35 dist-test-slave-j663 krb5kdc[28743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940035, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:20:35.350697 28827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.337644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:37189 (local address 127.11.123.190:46063)
0620 07:20:35.337782 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.337786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.338707 (+ 921us) server_negotiation.cc:407] Connection header received
0620 07:20:35.339972 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.339975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.340023 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.340097 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:35.341327 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.341973 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.343160 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.343390 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.343521 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.346368 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:35.346371 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:35.346373 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:35.346397 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:35.348180 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:35.348830 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:35.348838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:35.348839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:35.348934 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:35.349414 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:35.349417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:35.349419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:35.349634 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:35.349724 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.350458 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.350568 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:20:35.351991 28981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.338048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46063 (local address 127.11.123.129:37189)
0620 07:20:35.338546 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:35.338589 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:35.339768 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:35.340264 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:35.340270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:35.340650 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:35.341161 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.341175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.342114 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.342117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.343009 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.343018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.343559 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.343563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.343822 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.344521 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:20:35.344544 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:35.346190 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:35.348328 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:35.348336 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:35.348356 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:35.348718 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:35.349045 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:35.349051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:35.349056 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:35.349285 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:35.349763 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:35.349773 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:35.350150 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 07:20:35.350375 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 07:20:35.350769 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
I20260620 07:20:35.353454 28979 heartbeater.cc:347] Connected to a master server at 127.11.123.190:46063
I20260620 07:20:35.353744 28979 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:20:35.354450 28979 heartbeater.cc:511] Master 127.11.123.190:46063 requested a full tablet report, sending...
I20260620 07:20:35.356035 28788 ts_manager.cc:194] Registered new tserver with Master: 0ba086b4149040b497a2b006a82233d8 (127.11.123.129:41059)
I20260620 07:20:35.357329 28788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:37189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:20:35.377063 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46063
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:35.491323 28984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:35.491580 28984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:35.491664 28984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:35.495822 28984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:35.495882 28984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:35.495980 28984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:20:35.501417 28984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:35.502481 28984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:35.503502 28984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:35.513403 28990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:35.513392 28989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:35.513393 28992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:35.513612 28984 server_base.cc:1034] running on GCE node
I20260620 07:20:35.514123 28984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:35.514775 28984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:35.515976 28984 hybrid_clock.cc:648] HybridClock initialized: now 1781940035515961 us; error 27 us; skew 500 ppm
Jun 20 07:20:35 dist-test-slave-j663 krb5kdc[28743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940035, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:35.518702 28984 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:20:35.519582 28984 webserver.cc:460] Webserver started at http://127.11.123.130:44987/ using document root <none> and password file <none>
I20260620 07:20:35.520138 28984 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:35.520226 28984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:35.520457 28984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:35.522540 28984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "fecbc81223024fa78699117c69034317"
format_stamp: "Formatted at 2026-06-20 07:20:35 on dist-test-slave-j663"
server_key: "63a1599ec1768651933b4a85f05c66b0"
server_key_iv: "b3e90cafb90b42d546fdc1653f21e375"
server_key_version: "encryptionkey@0"
I20260620 07:20:35.523098 28984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "fecbc81223024fa78699117c69034317"
format_stamp: "Formatted at 2026-06-20 07:20:35 on dist-test-slave-j663"
server_key: "63a1599ec1768651933b4a85f05c66b0"
server_key_iv: "b3e90cafb90b42d546fdc1653f21e375"
server_key_version: "encryptionkey@0"
I20260620 07:20:35.526743 28984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:20:35.529064 28999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:35.530328 28984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:20:35.530462 28984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "fecbc81223024fa78699117c69034317"
format_stamp: "Formatted at 2026-06-20 07:20:35 on dist-test-slave-j663"
server_key: "63a1599ec1768651933b4a85f05c66b0"
server_key_iv: "b3e90cafb90b42d546fdc1653f21e375"
server_key_version: "encryptionkey@0"
I20260620 07:20:35.530570 28984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:35.555406 28984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:35.558576 28984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:35.558816 28984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:35.559403 28984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:35.560261 28984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:35.560331 28984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:35.560402 28984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:35.560452 28984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:35.570815 28984 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:35607
I20260620 07:20:35.570904 29112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:35607 every 8 connection(s)
I20260620 07:20:35.571734 28984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:20:35.573938 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 28984
I20260620 07:20:35.574067 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:20:35.574391 11758 external_mini_cluster.cc:1428] Setting key 498b73b4eb5cac7bb91160afda764c9a
Jun 20 07:20:35 dist-test-slave-j663 krb5kdc[28743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940035, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
I20260620 07:20:35.587298 28827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.573633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:56685 (local address 127.11.123.190:46063)
0620 07:20:35.573799 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.573803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.574871 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 07:20:35.576369 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.576376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.576438 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.576527 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:35.577891 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.578861 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.580139 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.580387 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.580509 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.583375 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:35.583380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:35.583384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:35.583422 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:35.585227 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:35.585822 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:35.585828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:35.585830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:35.585926 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:35.586244 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:35.586247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:35.586248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:35.586425 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:35.586580 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.587061 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.587171 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
I20260620 07:20:35.588297 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.574089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46063 (local address 127.11.123.130:56685)
0620 07:20:35.574674 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:35.574744 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:35.576160 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:35.576673 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:35.576680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:35.577059 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:35.577652 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.577668 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.578994 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.578997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.579995 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.580004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.580548 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.580553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.580776 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.581437 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:20:35.581453 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:35.583200 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:35.585343 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:35.585348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:35.585360 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:35.585709 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:35.586022 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:35.586025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:35.586026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:35.586155 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:35.586591 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:35.586601 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:35.586837 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 07:20:35.586995 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 07:20:35.587252 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":140,"threads_started":1}
I20260620 07:20:35.589529 29113 heartbeater.cc:347] Connected to a master server at 127.11.123.190:46063
I20260620 07:20:35.589758 29113 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:35.590245 29113 heartbeater.cc:511] Master 127.11.123.190:46063 requested a full tablet report, sending...
I20260620 07:20:35.591301 28788 ts_manager.cc:194] Registered new tserver with Master: fecbc81223024fa78699117c69034317 (127.11.123.130:35607)
I20260620 07:20:35.591894 28788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:56685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:20:35.603803 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46063
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:35.715855 29118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:35.716085 29118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:35.716125 29118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:35.720112 29118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:35.720166 29118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:35.720221 29118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:20:35.725210 29118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:46063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:35.726112 29118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:35.727030 29118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:35.736280 29123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:35.736366 29126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:35.736356 29124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:35.736553 29118 server_base.cc:1034] running on GCE node
I20260620 07:20:35.736994 29118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:35.737633 29118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:35.738829 29118 hybrid_clock.cc:648] HybridClock initialized: now 1781940035738802 us; error 43 us; skew 500 ppm
Jun 20 07:20:35 dist-test-slave-j663 krb5kdc[28743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940035, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:35.741411 29118 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:20:35.742219 29118 webserver.cc:460] Webserver started at http://127.11.123.131:33305/ using document root <none> and password file <none>
I20260620 07:20:35.742758 29118 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:35.742851 29118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:35.743067 29118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:35.744999 29118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "ddcab8ab92fc405790e7dbb76b27e6a7"
format_stamp: "Formatted at 2026-06-20 07:20:35 on dist-test-slave-j663"
server_key: "570b0e470fe810034da8531fd041d6c4"
server_key_iv: "3791a56b23bbfe1319179ad045620316"
server_key_version: "encryptionkey@0"
I20260620 07:20:35.745612 29118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "ddcab8ab92fc405790e7dbb76b27e6a7"
format_stamp: "Formatted at 2026-06-20 07:20:35 on dist-test-slave-j663"
server_key: "570b0e470fe810034da8531fd041d6c4"
server_key_iv: "3791a56b23bbfe1319179ad045620316"
server_key_version: "encryptionkey@0"
I20260620 07:20:35.749130 29118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:20:35.751384 29133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:35.752563 29118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:20:35.752702 29118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "ddcab8ab92fc405790e7dbb76b27e6a7"
format_stamp: "Formatted at 2026-06-20 07:20:35 on dist-test-slave-j663"
server_key: "570b0e470fe810034da8531fd041d6c4"
server_key_iv: "3791a56b23bbfe1319179ad045620316"
server_key_version: "encryptionkey@0"
I20260620 07:20:35.752807 29118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:35.778136 29118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:35.781554 29118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:35.781755 29118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:35.782335 29118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:35.783139 29118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:35.783185 29118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:35.783252 29118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:35.783288 29118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:35.792155 29118 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:40075
I20260620 07:20:35.792191 29246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:40075 every 8 connection(s)
I20260620 07:20:35.792994 29118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:20:35.799893 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 29118
I20260620 07:20:35.800024 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:20:35.800333 11758 external_mini_cluster.cc:1428] Setting key 7d21246d25c23a2967827935fa6bfcee
Jun 20 07:20:35 dist-test-slave-j663 krb5kdc[28743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940035, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:20:35.806589 28827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.794768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:34323 (local address 127.11.123.190:46063)
0620 07:20:35.794906 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.794910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.795743 (+ 833us) server_negotiation.cc:407] Connection header received
0620 07:20:35.797020 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.797023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.797130 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.797214 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:35.798305 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.798847 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.800177 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.800438 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.800533 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.803125 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:35.803128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:35.803131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:35.803153 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:35.804642 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:35.805060 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:35.805062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:35.805063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:35.805158 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:35.805487 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:35.805491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:35.805493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:35.805723 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:35.805821 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.806320 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.806463 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:20:35.807299 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.795143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:46063 (local address 127.11.123.131:34323)
0620 07:20:35.795565 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:35.795644 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:35.796806 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:35.797390 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:35.797397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:35.797720 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:35.798162 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.798175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.798974 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.798979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.800045 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.800057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.800580 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.800584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.800753 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.801444 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:20:35.801464 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:35.802982 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:35.804740 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:35.804743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:35.804754 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:35.804975 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:35.805239 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:35.805241 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:35.805242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:35.805364 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:35.805863 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:35.805868 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:35.806084 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 07:20:35.806237 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 07:20:35.806479 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
I20260620 07:20:35.808470 29247 heartbeater.cc:347] Connected to a master server at 127.11.123.190:46063
I20260620 07:20:35.808712 29247 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:35.809247 29247 heartbeater.cc:511] Master 127.11.123.190:46063 requested a full tablet report, sending...
I20260620 07:20:35.810302 28788 ts_manager.cc:194] Registered new tserver with Master: ddcab8ab92fc405790e7dbb76b27e6a7 (127.11.123.131:40075)
I20260620 07:20:35.810865 28788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:34323
I20260620 07:20:35.814109 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:20:35.822907 28827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.815938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.11.123.190:46063)
0620 07:20:35.816061 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.816066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.816229 (+ 163us) server_negotiation.cc:407] Connection header received
0620 07:20:35.816317 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.816319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.816354 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.816418 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:35.817061 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.817600 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.818433 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.818659 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.818741 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.819995 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:35.819999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:35.820001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:35.820022 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:35.821537 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:35.821912 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:35.821915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:35.821916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:35.821987 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:35.822217 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:35.822219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:35.822220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:35.822408 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:35.822546 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.822713 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.822800 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:20:35.825620 28788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:20:35.828469 28788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:20:35.842744 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.836736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:35607 (local address 127.0.0.1:45094)
0620 07:20:35.837281 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:35.837327 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:35.837462 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:35.838504 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:35.838507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:35.838525 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:35.838603 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.838609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.839821 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.839825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.841062 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.841093 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.842211 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.842215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.842314 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.842349 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:20:35.842433 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 07:20:35.842546 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":67,"spinlock_wait_cycles":4864,"thread_start_us":96,"threads_started":1}
I20260620 07:20:35.842952 29261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.837472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.11.123.129:41059)
0620 07:20:35.837836 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.837853 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.838189 (+ 336us) server_negotiation.cc:407] Connection header received
0620 07:20:35.838395 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.838402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.838579 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.838757 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:35.839138 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.840153 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.841370 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.842137 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.842238 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.842369 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.842688 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.842799 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260620 07:20:35.843626 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.836659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.11.123.131:40075)
0620 07:20:35.837060 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.837114 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.837641 (+ 527us) server_negotiation.cc:407] Connection header received
0620 07:20:35.837730 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.837735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.837950 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.838112 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:35.838417 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.839415 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.841251 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.842262 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.842408 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.842553 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.843300 (+ 747us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.843426 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":105,"threads_started":1}
I20260620 07:20:35.843192 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.836469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:40075 (local address 127.0.0.1:44254)
0620 07:20:35.837279 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:35.837327 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:35.837464 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:35.838118 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:35.838129 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:35.838157 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:35.838307 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.838317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.839565 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.839569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.841062 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.841075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.842640 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.842646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.842780 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.842815 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:20:35.842898 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:20:35.842989 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":51,"spinlock_wait_cycles":4992,"thread_start_us":100,"threads_started":1}
I20260620 07:20:35.843595 29263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.837790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:41059 (local address 127.0.0.1:50888)
0620 07:20:35.838092 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:35.838105 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:35.838250 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:35.838907 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:35.838911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:35.838928 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:35.839011 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.839016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.840309 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.840312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.841245 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.841252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.842465 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.842468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.842541 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.842558 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 07:20:35.843294 (+ 736us) client_negotiation.cc:240] Negotiation successful
0620 07:20:35.843366 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260620 07:20:35.844286 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.837679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.11.123.130:35607)
0620 07:20:35.838063 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.838074 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.838106 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:20:35.838176 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.838181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.838361 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.838507 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:35.838724 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.839653 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.841221 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.842064 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.843869 (+ 1805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.843988 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.844039 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.844135 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260620 07:20:35.847448 29181 tablet_service.cc:1467] Processing CreateTablet for tablet 9333d1bceeb64c97bd5501e756be72b4 (DEFAULT_TABLE table=test-table [id=f6e47a220bb34e71b4050eb3241a8c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:35.847328 29047 tablet_service.cc:1467] Processing CreateTablet for tablet 9333d1bceeb64c97bd5501e756be72b4 (DEFAULT_TABLE table=test-table [id=f6e47a220bb34e71b4050eb3241a8c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:35.848371 29181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9333d1bceeb64c97bd5501e756be72b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:35.848374 29047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9333d1bceeb64c97bd5501e756be72b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:35.850906 28913 tablet_service.cc:1467] Processing CreateTablet for tablet 9333d1bceeb64c97bd5501e756be72b4 (DEFAULT_TABLE table=test-table [id=f6e47a220bb34e71b4050eb3241a8c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:35.851749 28913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9333d1bceeb64c97bd5501e756be72b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:35.854961 29264 tablet_bootstrap.cc:492] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7: Bootstrap starting.
I20260620 07:20:35.857561 29264 tablet_bootstrap.cc:654] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:35.858117 29265 tablet_bootstrap.cc:492] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317: Bootstrap starting.
I20260620 07:20:35.858546 29266 tablet_bootstrap.cc:492] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8: Bootstrap starting.
I20260620 07:20:35.859061 29264 log.cc:824] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:35.860172 29265 tablet_bootstrap.cc:654] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:35.860539 29266 tablet_bootstrap.cc:654] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:35.861222 29264 tablet_bootstrap.cc:492] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7: No bootstrap required, opened a new log
I20260620 07:20:35.861255 29265 log.cc:824] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:35.861366 29266 log.cc:824] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:35.861418 29264 ts_tablet_manager.cc:1397] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 07:20:35.862883 29266 tablet_bootstrap.cc:492] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8: No bootstrap required, opened a new log
I20260620 07:20:35.863075 29266 ts_tablet_manager.cc:1397] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:20:35.863173 29265 tablet_bootstrap.cc:492] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317: No bootstrap required, opened a new log
I20260620 07:20:35.863363 29265 ts_tablet_manager.cc:1397] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:20:35.865223 29264 raft_consensus.cc:348] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40075 } } peers { permanent_uuid: "fecbc81223024fa78699117c69034317" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35607 } } peers { permanent_uuid: "0ba086b4149040b497a2b006a82233d8" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41059 } }
I20260620 07:20:35.865485 29264 raft_consensus.cc:374] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:35.865540 29264 raft_consensus.cc:729] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddcab8ab92fc405790e7dbb76b27e6a7, State: Initialized, Role: FOLLOWER
I20260620 07:20:35.866022 29264 consensus_queue.cc:260] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40075 } } peers { permanent_uuid: "fecbc81223024fa78699117c69034317" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35607 } } peers { permanent_uuid: "0ba086b4149040b497a2b006a82233d8" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41059 } }
I20260620 07:20:35.866837 29266 raft_consensus.cc:348] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40075 } } peers { permanent_uuid: "fecbc81223024fa78699117c69034317" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35607 } } peers { permanent_uuid: "0ba086b4149040b497a2b006a82233d8" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41059 } }
I20260620 07:20:35.867005 29264 ts_tablet_manager.cc:1428] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:20:35.867062 29266 raft_consensus.cc:374] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:35.867151 29266 raft_consensus.cc:729] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba086b4149040b497a2b006a82233d8, State: Initialized, Role: FOLLOWER
I20260620 07:20:35.867080 29265 raft_consensus.cc:348] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40075 } } peers { permanent_uuid: "fecbc81223024fa78699117c69034317" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35607 } } peers { permanent_uuid: "0ba086b4149040b497a2b006a82233d8" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41059 } }
I20260620 07:20:35.867324 29265 raft_consensus.cc:374] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:35.867347 29247 heartbeater.cc:503] Master 127.11.123.190:46063 was elected leader, sending a full tablet report...
I20260620 07:20:35.867393 29265 raft_consensus.cc:729] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fecbc81223024fa78699117c69034317, State: Initialized, Role: FOLLOWER
I20260620 07:20:35.867600 29266 consensus_queue.cc:260] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40075 } } peers { permanent_uuid: "fecbc81223024fa78699117c69034317" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35607 } } peers { permanent_uuid: "0ba086b4149040b497a2b006a82233d8" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41059 } }
I20260620 07:20:35.867774 29265 consensus_queue.cc:260] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40075 } } peers { permanent_uuid: "fecbc81223024fa78699117c69034317" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35607 } } peers { permanent_uuid: "0ba086b4149040b497a2b006a82233d8" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41059 } }
I20260620 07:20:35.868389 29113 heartbeater.cc:503] Master 127.11.123.190:46063 was elected leader, sending a full tablet report...
I20260620 07:20:35.868688 29265 ts_tablet_manager.cc:1428] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 07:20:35.869470 29266 ts_tablet_manager.cc:1428] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260620 07:20:35.869861 28979 heartbeater.cc:503] Master 127.11.123.190:46063 was elected leader, sending a full tablet report...
I20260620 07:20:35.947558 29271 raft_consensus.cc:482] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:20:35.947770 29271 raft_consensus.cc:504] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40075 } } peers { permanent_uuid: "fecbc81223024fa78699117c69034317" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35607 } } peers { permanent_uuid: "0ba086b4149040b497a2b006a82233d8" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41059 } }
I20260620 07:20:35.948953 29271 leader_election.cc:288] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddcab8ab92fc405790e7dbb76b27e6a7 (127.11.123.131:40075), fecbc81223024fa78699117c69034317 (127.11.123.130:35607)
I20260620 07:20:35.954206 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.949312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:35627 (local address 127.11.123.131:40075)
0620 07:20:35.949648 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.949654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.949673 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:20:35.949757 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.949759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.949801 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.949910 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:35.950244 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.951033 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.952954 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.953636 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.953742 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.953782 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.953990 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.954069 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 07:20:35.954248 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.949181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:40075 (local address 127.11.123.129:35627)
0620 07:20:35.949533 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:35.949549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:35.949662 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:35.949944 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:35.949948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:35.949967 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:35.950102 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.950111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.951144 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.951146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.952811 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.952822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.953733 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.953737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.953847 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.953875 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 07:20:35.953958 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:20:35.954049 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":184,"thread_start_us":133,"threads_started":1}
I20260620 07:20:35.954246 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.949357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:34095 (local address 127.11.123.130:35607)
0620 07:20:35.949670 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.949674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.949870 (+ 196us) server_negotiation.cc:407] Connection header received
0620 07:20:35.949967 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.949970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.950032 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.950101 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:35.950383 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.951110 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.952954 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.953594 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.953728 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.953782 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.954058 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.954140 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260620 07:20:35.954708 29274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.949243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:35607 (local address 127.11.123.129:34095)
0620 07:20:35.949740 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:35.949754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:35.949862 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:35.950137 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:35.950143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:35.950164 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:35.950282 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.950287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.951210 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.951212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.952819 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.952825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.953819 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.953824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.953910 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.953943 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:20:35.954378 (+ 435us) client_negotiation.cc:240] Negotiation successful
0620 07:20:35.954463 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":160,"thread_start_us":113,"threads_started":1}
I20260620 07:20:35.954897 29201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9333d1bceeb64c97bd5501e756be72b4" candidate_uuid: "0ba086b4149040b497a2b006a82233d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" is_pre_election: true
I20260620 07:20:35.955206 29201 raft_consensus.cc:2454] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ba086b4149040b497a2b006a82233d8 in term 0.
I20260620 07:20:35.955366 29272 raft_consensus.cc:482] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:20:35.955528 29272 raft_consensus.cc:504] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40075 } } peers { permanent_uuid: "fecbc81223024fa78699117c69034317" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35607 } } peers { permanent_uuid: "0ba086b4149040b497a2b006a82233d8" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41059 } }
I20260620 07:20:35.955643 29067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9333d1bceeb64c97bd5501e756be72b4" candidate_uuid: "0ba086b4149040b497a2b006a82233d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fecbc81223024fa78699117c69034317" is_pre_election: true
I20260620 07:20:35.955706 28868 leader_election.cc:302] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba086b4149040b497a2b006a82233d8, ddcab8ab92fc405790e7dbb76b27e6a7; no voters:
I20260620 07:20:35.955976 29271 raft_consensus.cc:2781] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:20:35.956012 29067 raft_consensus.cc:2454] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ba086b4149040b497a2b006a82233d8 in term 0.
I20260620 07:20:35.956075 29271 raft_consensus.cc:482] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:20:35.956120 29271 raft_consensus.cc:3037] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:35.957741 29271 raft_consensus.cc:504] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40075 } } peers { permanent_uuid: "fecbc81223024fa78699117c69034317" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35607 } } peers { permanent_uuid: "0ba086b4149040b497a2b006a82233d8" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41059 } }
I20260620 07:20:35.957965 29272 leader_election.cc:288] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddcab8ab92fc405790e7dbb76b27e6a7 (127.11.123.131:40075), 0ba086b4149040b497a2b006a82233d8 (127.11.123.129:41059)
I20260620 07:20:35.958098 29271 leader_election.cc:288] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [CANDIDATE]: Term 1 election: Requested vote from peers ddcab8ab92fc405790e7dbb76b27e6a7 (127.11.123.131:40075), fecbc81223024fa78699117c69034317 (127.11.123.130:35607)
I20260620 07:20:35.958365 29201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9333d1bceeb64c97bd5501e756be72b4" candidate_uuid: "0ba086b4149040b497a2b006a82233d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7"
I20260620 07:20:35.958493 29201 raft_consensus.cc:3037] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:35.959982 29201 raft_consensus.cc:2454] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ba086b4149040b497a2b006a82233d8 in term 1.
I20260620 07:20:35.958891 29067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9333d1bceeb64c97bd5501e756be72b4" candidate_uuid: "0ba086b4149040b497a2b006a82233d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fecbc81223024fa78699117c69034317"
I20260620 07:20:35.960251 29067 raft_consensus.cc:3037] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:35.960553 28868 leader_election.cc:302] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba086b4149040b497a2b006a82233d8, ddcab8ab92fc405790e7dbb76b27e6a7; no voters:
I20260620 07:20:35.960774 29271 raft_consensus.cc:2781] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:35.960983 29271 raft_consensus.cc:686] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [term 1 LEADER]: Becoming Leader. State: Replica: 0ba086b4149040b497a2b006a82233d8, State: Running, Role: LEADER
I20260620 07:20:35.961344 29271 consensus_queue.cc:237] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40075 } } peers { permanent_uuid: "fecbc81223024fa78699117c69034317" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35607 } } peers { permanent_uuid: "0ba086b4149040b497a2b006a82233d8" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41059 } }
I20260620 07:20:35.961741 29067 raft_consensus.cc:2454] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ba086b4149040b497a2b006a82233d8 in term 1.
I20260620 07:20:35.964872 29261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.957831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:59937 (local address 127.11.123.129:41059)
0620 07:20:35.957934 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.957939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.958154 (+ 215us) server_negotiation.cc:407] Connection header received
0620 07:20:35.958405 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.958408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.958461 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.958520 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:35.959316 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.960092 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.962273 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.962817 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.962904 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.962950 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.964687 (+ 1737us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.964765 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 07:20:35.965178 28788 catalog_manager.cc:5495] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ba086b4149040b497a2b006a82233d8 (127.11.123.129). New cstate: current_term: 1 leader_uuid: "0ba086b4149040b497a2b006a82233d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fecbc81223024fa78699117c69034317" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ba086b4149040b497a2b006a82233d8" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 41059 } health_report { overall_health: HEALTHY } } }
I20260620 07:20:35.965504 29275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.956979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:41059 (local address 127.11.123.130:59937)
0620 07:20:35.958014 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:35.958029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:35.958282 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:35.958717 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:35.958722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:35.958742 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:35.959165 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.959177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.960524 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.960527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.962118 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.962128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.963276 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.963279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.964114 (+ 835us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.964143 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 07:20:35.965289 (+ 1146us) client_negotiation.cc:240] Negotiation successful
0620 07:20:35.965367 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":555,"thread_start_us":117,"threads_started":1}
I20260620 07:20:35.966207 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.957250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:38583 (local address 127.11.123.131:40075)
0620 07:20:35.959111 (+ 1861us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.959115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.959129 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:20:35.959193 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.959196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.959236 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.959739 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:35.962171 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.963001 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.964568 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.965263 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.965939 (+ 676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.965983 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.966019 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.966085 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1763}
I20260620 07:20:35.965834 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.956967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:40075 (local address 127.11.123.130:38583)
0620 07:20:35.957162 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:35.957177 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:35.957458 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:35.959381 (+ 1923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:35.959384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:35.959402 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:35.959581 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.959589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.963150 (+ 3561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.963157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.964385 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:35.964397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.965401 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.965406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:35.965542 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.965572 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:20:35.965625 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 07:20:35.965714 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 07:20:35.966466 28933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9333d1bceeb64c97bd5501e756be72b4" candidate_uuid: "fecbc81223024fa78699117c69034317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba086b4149040b497a2b006a82233d8" is_pre_election: true
I20260620 07:20:35.966699 29201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9333d1bceeb64c97bd5501e756be72b4" candidate_uuid: "fecbc81223024fa78699117c69034317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" is_pre_election: true
I20260620 07:20:35.966850 29201 raft_consensus.cc:2379] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fecbc81223024fa78699117c69034317 in current term 1: Already voted for candidate 0ba086b4149040b497a2b006a82233d8 in this term.
I20260620 07:20:35.968024 29002 leader_election.cc:302] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fecbc81223024fa78699117c69034317; no voters: 0ba086b4149040b497a2b006a82233d8, ddcab8ab92fc405790e7dbb76b27e6a7
I20260620 07:20:35.968279 29272 raft_consensus.cc:2726] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 07:20:35.993731 29261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:35.990040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.11.123.129:41059)
0620 07:20:35.990188 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:35.990193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:35.990210 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:20:35.990298 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:35.990302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:35.990364 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:35.990471 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:20:35.990725 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.991238 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.992158 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:35.992384 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:35.992490 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:35.992814 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:20:35.993368 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:20:35.993496 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:35.993539 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 07:20:35.993589 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 07:20:36.000276 29067 raft_consensus.cc:1261] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [term 1 FOLLOWER]: Refusing update from remote peer 0ba086b4149040b497a2b006a82233d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:20:36.000277 29201 raft_consensus.cc:1261] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7 [term 1 FOLLOWER]: Refusing update from remote peer 0ba086b4149040b497a2b006a82233d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:20:36.000910 29276 consensus_queue.cc:1035] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fecbc81223024fa78699117c69034317" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:20:36.001189 29271 consensus_queue.cc:1035] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddcab8ab92fc405790e7dbb76b27e6a7" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:20:36.009559 29280 mvcc.cc:205] Tried to move back new op lower bound from 7298826387448950784 to 7298826387301298176. Current Snapshot: MvccSnapshot[applied={T|T < 7298826387448950784 or (T in {7298826387448950784})}]
I20260620 07:20:36.018237 28787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:20:36.018412 28787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:20:36.020790 28787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a311df401554fe58638ece739540a14: Sending DeleteTablet for 3 replicas of tablet 9333d1bceeb64c97bd5501e756be72b4
I20260620 07:20:36.021605 29181 tablet_service.cc:1514] Processing DeleteTablet for tablet 9333d1bceeb64c97bd5501e756be72b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:36 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:44254
I20260620 07:20:36.021739 29047 tablet_service.cc:1514] Processing DeleteTablet for tablet 9333d1bceeb64c97bd5501e756be72b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:36 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:45094
I20260620 07:20:36.021844 28913 tablet_service.cc:1514] Processing DeleteTablet for tablet 9333d1bceeb64c97bd5501e756be72b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:36 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:50888
I20260620 07:20:36.022202 29288 tablet_replica.cc:330] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7: stopping tablet replica
I20260620 07:20:36.022449 29290 tablet_replica.cc:330] T 9333d1bceeb64c97bd5501e756be72b4 P 0ba086b4149040b497a2b006a82233d8: stopping tablet replica
I20260620 07:20:36.022504 29288 raft_consensus.cc:2229] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:20:36.022827 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 28850
I20260620 07:20:36.022827 29288 raft_consensus.cc:2258] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:20:36.023528 29289 tablet_replica.cc:330] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317: stopping tablet replica
I20260620 07:20:36.023790 29289 raft_consensus.cc:2229] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:20:36.024045 29289 raft_consensus.cc:2258] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:20:36.025417 29288 ts_tablet_manager.cc:1905] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:20:36.025600 29289 ts_tablet_manager.cc:1905] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:20:36.027683 29288 ts_tablet_manager.cc:1918] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:20:36.027791 29288 log.cc:1196] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/9333d1bceeb64c97bd5501e756be72b4
I20260620 07:20:36.028110 29288 ts_tablet_manager.cc:1939] T 9333d1bceeb64c97bd5501e756be72b4 P ddcab8ab92fc405790e7dbb76b27e6a7: Deleting consensus metadata
I20260620 07:20:36.028931 29289 ts_tablet_manager.cc:1918] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:20:36.029033 29289 log.cc:1196] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/9333d1bceeb64c97bd5501e756be72b4
I20260620 07:20:36.029266 28774 catalog_manager.cc:4841] TS ddcab8ab92fc405790e7dbb76b27e6a7 (127.11.123.131:40075): tablet 9333d1bceeb64c97bd5501e756be72b4 (table test-table [id=f6e47a220bb34e71b4050eb3241a8c68]) successfully deleted
I20260620 07:20:36.029493 29289 ts_tablet_manager.cc:1939] T 9333d1bceeb64c97bd5501e756be72b4 P fecbc81223024fa78699117c69034317: Deleting consensus metadata
I20260620 07:20:36.030730 28775 catalog_manager.cc:4841] TS fecbc81223024fa78699117c69034317 (127.11.123.130:35607): tablet 9333d1bceeb64c97bd5501e756be72b4 (table test-table [id=f6e47a220bb34e71b4050eb3241a8c68]) successfully deleted
W20260620 07:20:36.031327 28775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:41059 (error 108)
I20260620 07:20:36.031368 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 28984
I20260620 07:20:36.032644 29263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.032392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:41059 (local address 127.0.0.1:50894)
0620 07:20:36.032507 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:36.032572 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 07:20:36.032783 28775 catalog_manager.cc:4584] TS 0ba086b4149040b497a2b006a82233d8 (127.11.123.129:41059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9333d1bceeb64c97bd5501e756be72b4: Network error: Client connection negotiation failed: client connection to 127.11.123.129:41059: connect: Connection refused (error 111)
I20260620 07:20:36.038580 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 29118
I20260620 07:20:36.044947 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 28756
2026-06-20T07:20:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2393 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T07:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:36Z Disabled control of system clock
I20260620 07:20:36.076598 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:34363
--webserver_interface=127.11.123.190
--webserver_port=0
--builtin_ntp_servers=127.11.123.148:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:34363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:20:36.181826 29295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:36.182109 29295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:36.182169 29295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:36.186900 29295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:36.186982 29295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:36.187026 29295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:36.187065 29295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:36.187119 29295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:36.191794 29295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:34363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:34363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:36.192812 29295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:36.194164 29295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:36.202299 29301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:36.202338 29303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:36.202325 29300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:36.202405 29295 server_base.cc:1034] running on GCE node
I20260620 07:20:36.203006 29295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:36.204072 29295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:36.205264 29295 hybrid_clock.cc:648] HybridClock initialized: now 1781940036205250 us; error 34 us; skew 500 ppm
I20260620 07:20:36.207063 29295 webserver.cc:460] Webserver started at http://127.11.123.190:43225/ using document root <none> and password file <none>
I20260620 07:20:36.207628 29295 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:36.207723 29295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:36.207950 29295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:36.209972 29295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "ebe255045d6747db9cb351885a8c172a"
format_stamp: "Formatted at 2026-06-20 07:20:36 on dist-test-slave-j663"
server_key: "ac78775ea28f7c97ed8669ea292f9e57"
server_key_iv: "018741ee43c85974532e82b93793d7c4"
server_key_version: "encryptionkey@0"
I20260620 07:20:36.210525 29295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "ebe255045d6747db9cb351885a8c172a"
format_stamp: "Formatted at 2026-06-20 07:20:36 on dist-test-slave-j663"
server_key: "ac78775ea28f7c97ed8669ea292f9e57"
server_key_iv: "018741ee43c85974532e82b93793d7c4"
server_key_version: "encryptionkey@0"
I20260620 07:20:36.214067 29295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:20:36.217273 29309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:36.218462 29295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:20:36.218566 29295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "ebe255045d6747db9cb351885a8c172a"
format_stamp: "Formatted at 2026-06-20 07:20:36 on dist-test-slave-j663"
server_key: "ac78775ea28f7c97ed8669ea292f9e57"
server_key_iv: "018741ee43c85974532e82b93793d7c4"
server_key_version: "encryptionkey@0"
I20260620 07:20:36.218647 29295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:36.230469 29295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:36.231029 29295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:36.231235 29295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:36.238091 29295 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:34363
I20260620 07:20:36.238109 29361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:34363 every 8 connection(s)
I20260620 07:20:36.239010 29295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:36.241384 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 29295
I20260620 07:20:36.241514 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:36.241837 11758 external_mini_cluster.cc:1428] Setting key 86525d7488a556bdc7ac43c00305b47d
I20260620 07:20:36.242318 29362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:36.249114 29362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a: Bootstrap starting.
I20260620 07:20:36.251847 29362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:36.251965 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.243207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.11.123.190:34363)
0620 07:20:36.243812 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:36.243828 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:36.243862 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:20:36.244868 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:36.244886 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:36.245340 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:36.245615 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:36.245936 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.246961 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.248047 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.248731 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.248887 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.249922 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:36.249929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:36.249942 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:36.250444 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:36.250516 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:36.250542 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:36.250642 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:36.250815 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 07:20:36.251071 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":145,"threads_started":1}
I20260620 07:20:36.252712 29362 log.cc:824] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:36.255817 29362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a: No bootstrap required, opened a new log
I20260620 07:20:36.259402 29362 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe255045d6747db9cb351885a8c172a" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34363 } }
I20260620 07:20:36.259632 29362 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:36.259712 29362 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe255045d6747db9cb351885a8c172a, State: Initialized, Role: FOLLOWER
I20260620 07:20:36.260142 29362 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe255045d6747db9cb351885a8c172a" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34363 } }
I20260620 07:20:36.260282 29362 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:36.260344 29362 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:36.260452 29362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:36.261356 29362 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe255045d6747db9cb351885a8c172a" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34363 } }
I20260620 07:20:36.261689 29362 leader_election.cc:302] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebe255045d6747db9cb351885a8c172a; no voters:
I20260620 07:20:36.262014 29362 leader_election.cc:288] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:36.262125 29367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:36.262336 29367 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [term 1 LEADER]: Becoming Leader. State: Replica: ebe255045d6747db9cb351885a8c172a, State: Running, Role: LEADER
I20260620 07:20:36.262626 29367 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe255045d6747db9cb351885a8c172a" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34363 } }
I20260620 07:20:36.263236 29362 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:36.264348 29369 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe255045d6747db9cb351885a8c172a. Latest consensus state: current_term: 1 leader_uuid: "ebe255045d6747db9cb351885a8c172a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe255045d6747db9cb351885a8c172a" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34363 } } }
I20260620 07:20:36.264472 29369 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:36.264724 29368 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebe255045d6747db9cb351885a8c172a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe255045d6747db9cb351885a8c172a" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34363 } } }
I20260620 07:20:36.264813 29368 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:36.265368 29376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:36.268265 29376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:36.273491 29376 catalog_manager.cc:1347] Generated new cluster ID: 411a3f73e52f41c1a518ebfb9a947f2f
I20260620 07:20:36.273813 29376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:36.293462 29376 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:36.294656 29376 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:36.300791 29376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a: Generated new TSK 0
I20260620 07:20:36.301591 29376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:36.315104 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34363
--builtin_ntp_servers=127.11.123.148:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:20:36.426450 29386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:36.426692 29386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:36.426729 29386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:36.430594 29386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:36.430639 29386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:36.430691 29386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:20:36.435287 29386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:36.436225 29386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:36.437608 29386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:36.446839 29392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:36.447042 29386 server_base.cc:1034] running on GCE node
W20260620 07:20:36.446851 29394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:36.446854 29391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:36.447510 29386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:36.448108 29386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:36.449292 29386 hybrid_clock.cc:648] HybridClock initialized: now 1781940036449261 us; error 43 us; skew 500 ppm
I20260620 07:20:36.451099 29386 webserver.cc:460] Webserver started at http://127.11.123.129:44493/ using document root <none> and password file <none>
I20260620 07:20:36.451654 29386 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:36.451747 29386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:36.451977 29386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:36.454536 29386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "7ed993b650504f72b32b94b632c62e8b"
format_stamp: "Formatted at 2026-06-20 07:20:36 on dist-test-slave-j663"
server_key: "d4fe2c32b1d360d55768db508f4b3718"
server_key_iv: "1ed916c60ba3e8cbcca1d4c5de2b6800"
server_key_version: "encryptionkey@0"
I20260620 07:20:36.455291 29386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "7ed993b650504f72b32b94b632c62e8b"
format_stamp: "Formatted at 2026-06-20 07:20:36 on dist-test-slave-j663"
server_key: "d4fe2c32b1d360d55768db508f4b3718"
server_key_iv: "1ed916c60ba3e8cbcca1d4c5de2b6800"
server_key_version: "encryptionkey@0"
I20260620 07:20:36.459915 29386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:20:36.462628 29400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:36.463912 29386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:20:36.464010 29386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "7ed993b650504f72b32b94b632c62e8b"
format_stamp: "Formatted at 2026-06-20 07:20:36 on dist-test-slave-j663"
server_key: "d4fe2c32b1d360d55768db508f4b3718"
server_key_iv: "1ed916c60ba3e8cbcca1d4c5de2b6800"
server_key_version: "encryptionkey@0"
I20260620 07:20:36.464138 29386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:36.478737 29386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:36.479385 29386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:36.479594 29386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:36.480170 29386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:36.481058 29386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:36.481184 29386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:36.481246 29386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:36.481278 29386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:36.490243 29386 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:34299
I20260620 07:20:36.490356 29513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:34299 every 8 connection(s)
I20260620 07:20:36.491140 29386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:20:36.500654 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.492960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:59167 (local address 127.11.123.190:34363)
0620 07:20:36.493143 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:36.493147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:36.494174 (+ 1027us) server_negotiation.cc:407] Connection header received
0620 07:20:36.495609 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:36.495612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:36.495678 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:36.495751 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:36.496614 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.497568 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.498793 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.499028 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.499141 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.500134 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:36.500138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:36.500140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:36.500153 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:36.500173 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:36.500190 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:36.500266 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:36.500430 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 07:20:36.500504 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:20:36.501224 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 29386
I20260620 07:20:36.501355 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:20:36.501566 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.493409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:34363 (local address 127.11.123.129:59167)
0620 07:20:36.494026 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:36.494063 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:36.495396 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:36.495909 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:36.495916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:36.495995 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:36.496448 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.496459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.497734 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.497737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.498666 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.498676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.499182 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.499187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.499346 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.499890 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:20:36.499892 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:36.499929 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:20:36.499985 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:36.500280 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:36.500293 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:36.500302 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:20:36.500417 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 07:20:36.500670 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
I20260620 07:20:36.501725 11758 external_mini_cluster.cc:1428] Setting key fed406189bf94aff7d42f17aa5611d32
I20260620 07:20:36.502779 29514 heartbeater.cc:347] Connected to a master server at 127.11.123.190:34363
I20260620 07:20:36.503010 29514 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:36.503492 29514 heartbeater.cc:511] Master 127.11.123.190:34363 requested a full tablet report, sending...
I20260620 07:20:36.503639 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34363
--builtin_ntp_servers=127.11.123.148:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:20:36.504961 29325 ts_manager.cc:194] Registered new tserver with Master: 7ed993b650504f72b32b94b632c62e8b (127.11.123.129:34299)
I20260620 07:20:36.506062 29325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.123.129:59167
W20260620 07:20:36.615327 29517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:36.615557 29517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:36.615593 29517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:36.619585 29517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:36.619634 29517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:36.619686 29517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:20:36.624295 29517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:36.625355 29517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:36.626699 29517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:36.636521 29525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:36.636511 29523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:36.636511 29522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:36.636734 29517 server_base.cc:1034] running on GCE node
I20260620 07:20:36.637243 29517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:36.637892 29517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:36.639056 29517 hybrid_clock.cc:648] HybridClock initialized: now 1781940036639039 us; error 31 us; skew 500 ppm
I20260620 07:20:36.640893 29517 webserver.cc:460] Webserver started at http://127.11.123.130:41773/ using document root <none> and password file <none>
I20260620 07:20:36.641521 29517 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:36.641597 29517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:36.641810 29517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:36.643783 29517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "fb05d7f12b4241acb298cc040bbcc38f"
format_stamp: "Formatted at 2026-06-20 07:20:36 on dist-test-slave-j663"
server_key: "a78fc2d9595867f44c61404b4854f0a0"
server_key_iv: "1bc19aedda3c000e104bff6aa3727a56"
server_key_version: "encryptionkey@0"
I20260620 07:20:36.644336 29517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "fb05d7f12b4241acb298cc040bbcc38f"
format_stamp: "Formatted at 2026-06-20 07:20:36 on dist-test-slave-j663"
server_key: "a78fc2d9595867f44c61404b4854f0a0"
server_key_iv: "1bc19aedda3c000e104bff6aa3727a56"
server_key_version: "encryptionkey@0"
I20260620 07:20:36.648162 29517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:20:36.650770 29531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:36.652070 29517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:20:36.652169 29517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "fb05d7f12b4241acb298cc040bbcc38f"
format_stamp: "Formatted at 2026-06-20 07:20:36 on dist-test-slave-j663"
server_key: "a78fc2d9595867f44c61404b4854f0a0"
server_key_iv: "1bc19aedda3c000e104bff6aa3727a56"
server_key_version: "encryptionkey@0"
I20260620 07:20:36.652297 29517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:36.667657 29517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:36.668293 29517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:36.668476 29517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:36.669024 29517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:36.669965 29517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:36.670012 29517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:36.670049 29517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:36.670065 29517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:36.678622 29517 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:42431
I20260620 07:20:36.678689 29644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:42431 every 8 connection(s)
I20260620 07:20:36.679570 29517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:20:36.688283 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.681369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:43951 (local address 127.11.123.190:34363)
0620 07:20:36.681540 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:36.681545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:36.682547 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 07:20:36.683882 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:36.683885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:36.683930 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:36.684012 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:36.684857 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.685455 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.686597 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.686814 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.686897 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.687890 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:36.687893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:36.687895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:36.687906 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:36.687924 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:36.687932 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:36.687995 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:36.688127 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 07:20:36.688174 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:20:36.689069 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.681774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:34363 (local address 127.11.123.130:43951)
0620 07:20:36.682398 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:36.682436 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:36.683661 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:36.684174 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:36.684180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:36.684259 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:36.684722 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.684733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.685615 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.685618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.686469 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.686478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.686946 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.686949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.687082 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.687647 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:20:36.687648 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:36.687686 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:20:36.687744 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:36.688006 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:36.688012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:36.688016 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:20:36.688113 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 07:20:36.688310 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":115,"threads_started":1}
I20260620 07:20:36.689682 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 29517
I20260620 07:20:36.689826 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:20:36.690115 11758 external_mini_cluster.cc:1428] Setting key 8da5e8f373724dde664b6a61627eda8a
I20260620 07:20:36.690316 29645 heartbeater.cc:347] Connected to a master server at 127.11.123.190:34363
I20260620 07:20:36.690559 29645 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:36.691123 29645 heartbeater.cc:511] Master 127.11.123.190:34363 requested a full tablet report, sending...
I20260620 07:20:36.691965 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34363
--builtin_ntp_servers=127.11.123.148:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:20:36.692128 29325 ts_manager.cc:194] Registered new tserver with Master: fb05d7f12b4241acb298cc040bbcc38f (127.11.123.130:42431)
I20260620 07:20:36.692690 29325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.123.130:43951
W20260620 07:20:36.803591 29648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:36.803809 29648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:36.803845 29648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:36.807860 29648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:36.807904 29648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:36.807996 29648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:20:36.812575 29648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:36.813585 29648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:36.814862 29648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:36.824154 29653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:36.824180 29656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:36.824180 29654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:36.824406 29648 server_base.cc:1034] running on GCE node
I20260620 07:20:36.824952 29648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:36.825701 29648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:36.826869 29648 hybrid_clock.cc:648] HybridClock initialized: now 1781940036826856 us; error 25 us; skew 500 ppm
I20260620 07:20:36.828500 29648 webserver.cc:460] Webserver started at http://127.11.123.131:36667/ using document root <none> and password file <none>
I20260620 07:20:36.829064 29648 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:36.829216 29648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:36.829499 29648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:36.831696 29648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "f51ad2e7fa7748f1a01ab5830f3338dd"
format_stamp: "Formatted at 2026-06-20 07:20:36 on dist-test-slave-j663"
server_key: "5c8cc6e67c9ae90f4dc2f88960e9c412"
server_key_iv: "e9fc20090019e109627358a11fba21f5"
server_key_version: "encryptionkey@0"
I20260620 07:20:36.832223 29648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "f51ad2e7fa7748f1a01ab5830f3338dd"
format_stamp: "Formatted at 2026-06-20 07:20:36 on dist-test-slave-j663"
server_key: "5c8cc6e67c9ae90f4dc2f88960e9c412"
server_key_iv: "e9fc20090019e109627358a11fba21f5"
server_key_version: "encryptionkey@0"
I20260620 07:20:36.835800 29648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 07:20:36.838366 29662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:36.839509 29648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:20:36.839641 29648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "f51ad2e7fa7748f1a01ab5830f3338dd"
format_stamp: "Formatted at 2026-06-20 07:20:36 on dist-test-slave-j663"
server_key: "5c8cc6e67c9ae90f4dc2f88960e9c412"
server_key_iv: "e9fc20090019e109627358a11fba21f5"
server_key_version: "encryptionkey@0"
I20260620 07:20:36.839746 29648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:36.855049 29648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:36.855635 29648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:36.855813 29648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:36.856381 29648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:36.857339 29648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:36.857406 29648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:36.857474 29648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:36.857525 29648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:36.868237 29648 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:41457
I20260620 07:20:36.868292 29775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:41457 every 8 connection(s)
I20260620 07:20:36.869294 29648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:20:36.878046 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.871215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:50161 (local address 127.11.123.190:34363)
0620 07:20:36.871359 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:36.871363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:36.872400 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 07:20:36.873759 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:36.873762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:36.873808 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:36.873879 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:36.874785 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.875290 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.876341 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.876540 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.876611 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.877669 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:36.877672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:36.877674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:36.877684 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:36.877701 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:36.877708 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:36.877764 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:36.877891 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 07:20:36.877936 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:20:36.878198 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 29648
I20260620 07:20:36.878304 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:20:36.878595 11758 external_mini_cluster.cc:1428] Setting key 76a6eccc56b0c32567e8d2a34ac3ee38
I20260620 07:20:36.879228 29778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.871644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:34363 (local address 127.11.123.131:50161)
0620 07:20:36.872259 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:36.872297 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:36.873576 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:36.874033 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:36.874039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:36.874118 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:36.874636 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.874655 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.875393 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.875395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.876243 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.876252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.876657 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.876660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.876786 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.877438 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:20:36.877440 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:36.877477 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:20:36.877535 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:36.877782 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:36.877788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:36.877792 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:20:36.877879 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:20:36.878285 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260620 07:20:36.880541 29776 heartbeater.cc:347] Connected to a master server at 127.11.123.190:34363
I20260620 07:20:36.880786 29776 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:36.881304 29776 heartbeater.cc:511] Master 127.11.123.190:34363 requested a full tablet report, sending...
I20260620 07:20:36.882207 29326 ts_manager.cc:194] Registered new tserver with Master: f51ad2e7fa7748f1a01ab5830f3338dd (127.11.123.131:41457)
I20260620 07:20:36.882762 29326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.123.131:50161
I20260620 07:20:36.892113 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:20:36.897712 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.893976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.11.123.190:34363)
0620 07:20:36.894125 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:36.894130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:36.894297 (+ 167us) server_negotiation.cc:407] Connection header received
0620 07:20:36.894428 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:36.894431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:36.894478 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:36.894562 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:36.894853 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.895351 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.896193 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.896434 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.896547 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.897321 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:36.897327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:36.897334 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:36.897345 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:36.897360 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:36.897367 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:36.897444 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:36.897532 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 07:20:36.897578 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:20:36.900276 29326 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:20:36.903350 29326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:20:36.917709 29790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.911898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:34299 (local address 127.0.0.1:49082)
0620 07:20:36.912531 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:36.912544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:36.912633 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:36.913383 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:36.913385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:36.913400 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:36.913491 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.913496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.914845 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.914848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.915964 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.915973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.917179 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.917184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.917298 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.917338 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 07:20:36.917427 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 07:20:36.917544 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260620 07:20:36.918423 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.912287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.11.123.129:34299)
0620 07:20:36.912889 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:36.912903 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:36.912928 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:20:36.913007 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:36.913012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:36.913237 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:36.913401 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:36.913604 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.914722 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.916081 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.917001 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.917923 (+ 922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.918110 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:36.918166 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 07:20:36.918261 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":138,"threads_started":1}
I20260620 07:20:36.919229 29787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.911402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:42431 (local address 127.0.0.1:47868)
0620 07:20:36.912349 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:36.912399 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:36.912531 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:36.913094 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:36.913102 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:36.913125 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:36.913264 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.913271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.914441 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.914447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.916934 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.916943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.918715 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.918721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.918838 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.918870 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:20:36.918942 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 07:20:36.919030 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":1216,"thread_start_us":95,"threads_started":1}
I20260620 07:20:36.919976 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.911904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.11.123.130:42431)
0620 07:20:36.912407 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:36.912421 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:36.912696 (+ 275us) server_negotiation.cc:407] Connection header received
0620 07:20:36.912765 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:36.912769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:36.912910 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:36.913051 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:36.913605 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.914325 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.917059 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.918179 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.918312 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.919606 (+ 1294us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:36.919662 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 07:20:36.919778 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":136,"threads_started":1}
I20260620 07:20:36.920677 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.911386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.11.123.131:41457)
0620 07:20:36.911995 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:36.912010 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:36.912488 (+ 478us) server_negotiation.cc:407] Connection header received
0620 07:20:36.912704 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:36.912710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:36.912910 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:36.913041 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:36.913495 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.914722 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.916878 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.919801 (+ 2923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.919921 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.920070 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:36.920412 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 07:20:36.920511 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":103,"threads_started":1}
I20260620 07:20:36.921008 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.911238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:41457 (local address 127.0.0.1:58642)
0620 07:20:36.912349 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:36.912399 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:36.912542 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:36.913246 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:36.913248 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:36.913262 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:36.913374 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.913380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.915705 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.915708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.916727 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.916741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.920137 (+ 3396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.920141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.920250 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.920273 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:20:36.920810 (+ 537us) client_negotiation.cc:240] Negotiation successful
0620 07:20:36.920892 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":71,"threads_started":1}
I20260620 07:20:36.922492 29448 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51 (DEFAULT_TABLE table=test-table [id=ddd8558d830742d6b719562932cd3f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:36.923025 29579 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51 (DEFAULT_TABLE table=test-table [id=ddd8558d830742d6b719562932cd3f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:36.923596 29448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:36.924008 29579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:36.924225 29710 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51 (DEFAULT_TABLE table=test-table [id=ddd8558d830742d6b719562932cd3f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:36.925176 29710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:36.932216 29795 tablet_bootstrap.cc:492] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd: Bootstrap starting.
I20260620 07:20:36.933426 29794 tablet_bootstrap.cc:492] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f: Bootstrap starting.
I20260620 07:20:36.934379 29795 tablet_bootstrap.cc:654] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:36.935230 29795 log.cc:824] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:36.935493 29794 tablet_bootstrap.cc:654] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:36.935969 29793 tablet_bootstrap.cc:492] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b: Bootstrap starting.
I20260620 07:20:36.936326 29794 log.cc:824] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:36.937880 29793 tablet_bootstrap.cc:654] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:36.938252 29794 tablet_bootstrap.cc:492] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f: No bootstrap required, opened a new log
I20260620 07:20:36.938480 29794 ts_tablet_manager.cc:1397] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:20:36.938690 29793 log.cc:824] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:36.939729 29795 tablet_bootstrap.cc:492] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd: No bootstrap required, opened a new log
I20260620 07:20:36.939914 29795 ts_tablet_manager.cc:1397] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 07:20:36.940371 29793 tablet_bootstrap.cc:492] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b: No bootstrap required, opened a new log
I20260620 07:20:36.940554 29793 ts_tablet_manager.cc:1397] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:20:36.943104 29794 raft_consensus.cc:348] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 41457 } } peers { permanent_uuid: "fb05d7f12b4241acb298cc040bbcc38f" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42431 } } peers { permanent_uuid: "7ed993b650504f72b32b94b632c62e8b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34299 } }
I20260620 07:20:36.943404 29794 raft_consensus.cc:374] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:36.943552 29794 raft_consensus.cc:729] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb05d7f12b4241acb298cc040bbcc38f, State: Initialized, Role: FOLLOWER
I20260620 07:20:36.944028 29794 consensus_queue.cc:260] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 41457 } } peers { permanent_uuid: "fb05d7f12b4241acb298cc040bbcc38f" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42431 } } peers { permanent_uuid: "7ed993b650504f72b32b94b632c62e8b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34299 } }
I20260620 07:20:36.944075 29793 raft_consensus.cc:348] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 41457 } } peers { permanent_uuid: "fb05d7f12b4241acb298cc040bbcc38f" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42431 } } peers { permanent_uuid: "7ed993b650504f72b32b94b632c62e8b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34299 } }
I20260620 07:20:36.944324 29793 raft_consensus.cc:374] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:36.944169 29795 raft_consensus.cc:348] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 41457 } } peers { permanent_uuid: "fb05d7f12b4241acb298cc040bbcc38f" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42431 } } peers { permanent_uuid: "7ed993b650504f72b32b94b632c62e8b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34299 } }
I20260620 07:20:36.944393 29793 raft_consensus.cc:729] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed993b650504f72b32b94b632c62e8b, State: Initialized, Role: FOLLOWER
I20260620 07:20:36.944406 29795 raft_consensus.cc:374] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:36.944453 29795 raft_consensus.cc:729] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f51ad2e7fa7748f1a01ab5830f3338dd, State: Initialized, Role: FOLLOWER
I20260620 07:20:36.944805 29793 consensus_queue.cc:260] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 41457 } } peers { permanent_uuid: "fb05d7f12b4241acb298cc040bbcc38f" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42431 } } peers { permanent_uuid: "7ed993b650504f72b32b94b632c62e8b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34299 } }
I20260620 07:20:36.944876 29795 consensus_queue.cc:260] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 41457 } } peers { permanent_uuid: "fb05d7f12b4241acb298cc040bbcc38f" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42431 } } peers { permanent_uuid: "7ed993b650504f72b32b94b632c62e8b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34299 } }
I20260620 07:20:36.945025 29794 ts_tablet_manager.cc:1428] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260620 07:20:36.945393 29645 heartbeater.cc:503] Master 127.11.123.190:34363 was elected leader, sending a full tablet report...
I20260620 07:20:36.945438 29514 heartbeater.cc:503] Master 127.11.123.190:34363 was elected leader, sending a full tablet report...
I20260620 07:20:36.945724 29793 ts_tablet_manager.cc:1428] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 07:20:36.946342 29776 heartbeater.cc:503] Master 127.11.123.190:34363 was elected leader, sending a full tablet report...
I20260620 07:20:36.947350 29795 ts_tablet_manager.cc:1428] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 07:20:36.970255 29800 raft_consensus.cc:482] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:20:36.970464 29800 raft_consensus.cc:504] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 41457 } } peers { permanent_uuid: "fb05d7f12b4241acb298cc040bbcc38f" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42431 } } peers { permanent_uuid: "7ed993b650504f72b32b94b632c62e8b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34299 } }
I20260620 07:20:36.971589 29800 leader_election.cc:288] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f51ad2e7fa7748f1a01ab5830f3338dd (127.11.123.131:41457), fb05d7f12b4241acb298cc040bbcc38f (127.11.123.130:42431)
I20260620 07:20:36.976466 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.971897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:35449 (local address 127.11.123.130:42431)
0620 07:20:36.972018 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:36.972026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:36.972045 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:20:36.972172 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:36.972175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:36.972227 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:36.972330 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:36.972639 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.973437 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.975192 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.975790 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.975915 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.975961 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:36.976271 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 07:20:36.976341 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:20:36.976466 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.971897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:52217 (local address 127.11.123.131:41457)
0620 07:20:36.972043 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:36.972047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:36.972294 (+ 247us) server_negotiation.cc:407] Connection header received
0620 07:20:36.972417 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:36.972420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:36.972465 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:36.972536 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:36.972803 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.973538 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.975192 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.975797 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.975915 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.975961 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:36.976271 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 07:20:36.976340 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 07:20:36.976532 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.971804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:42431 (local address 127.11.123.129:35449)
0620 07:20:36.971963 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:36.971979 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:36.972065 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:36.972357 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:36.972363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:36.972382 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:36.972498 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.972503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.973560 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.973564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.975042 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.975063 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.975918 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.975922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.976105 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.976165 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 07:20:36.976239 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 07:20:36.976360 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":177}
I20260620 07:20:36.976540 29802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:36.971804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:41457 (local address 127.11.123.129:52217)
0620 07:20:36.972198 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:36.972211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:36.972290 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:36.972569 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:36.972574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:36.972593 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:36.972705 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.972710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.973641 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.973645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.975042 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:36.975053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:36.975918 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:36.975922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:36.976081 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:36.976159 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 07:20:36.976239 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:20:36.976359 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":309,"thread_start_us":98,"threads_started":1}
I20260620 07:20:36.977209 29599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3aa1eb88a745dc9da9b6b9c03dbe51" candidate_uuid: "7ed993b650504f72b32b94b632c62e8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb05d7f12b4241acb298cc040bbcc38f" is_pre_election: true
I20260620 07:20:36.977245 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3aa1eb88a745dc9da9b6b9c03dbe51" candidate_uuid: "7ed993b650504f72b32b94b632c62e8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd" is_pre_election: true
I20260620 07:20:36.977502 29599 raft_consensus.cc:2454] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed993b650504f72b32b94b632c62e8b in term 0.
I20260620 07:20:36.977510 29730 raft_consensus.cc:2454] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed993b650504f72b32b94b632c62e8b in term 0.
I20260620 07:20:36.978011 29404 leader_election.cc:302] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7ed993b650504f72b32b94b632c62e8b, f51ad2e7fa7748f1a01ab5830f3338dd, fb05d7f12b4241acb298cc040bbcc38f; no voters:
I20260620 07:20:36.978303 29800 raft_consensus.cc:2781] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:20:36.978379 29800 raft_consensus.cc:482] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:20:36.978430 29800 raft_consensus.cc:3037] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:36.979408 29800 raft_consensus.cc:504] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 41457 } } peers { permanent_uuid: "fb05d7f12b4241acb298cc040bbcc38f" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42431 } } peers { permanent_uuid: "7ed993b650504f72b32b94b632c62e8b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34299 } }
I20260620 07:20:36.979782 29800 leader_election.cc:288] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [CANDIDATE]: Term 1 election: Requested vote from peers f51ad2e7fa7748f1a01ab5830f3338dd (127.11.123.131:41457), fb05d7f12b4241acb298cc040bbcc38f (127.11.123.130:42431)
I20260620 07:20:36.980137 29599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3aa1eb88a745dc9da9b6b9c03dbe51" candidate_uuid: "7ed993b650504f72b32b94b632c62e8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb05d7f12b4241acb298cc040bbcc38f"
I20260620 07:20:36.980269 29599 raft_consensus.cc:3037] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:36.980288 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3aa1eb88a745dc9da9b6b9c03dbe51" candidate_uuid: "7ed993b650504f72b32b94b632c62e8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd"
I20260620 07:20:36.980391 29730 raft_consensus.cc:3037] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:36.981567 29599 raft_consensus.cc:2454] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed993b650504f72b32b94b632c62e8b in term 1.
I20260620 07:20:36.981559 29730 raft_consensus.cc:2454] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed993b650504f72b32b94b632c62e8b in term 1.
I20260620 07:20:36.981905 29403 leader_election.cc:302] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed993b650504f72b32b94b632c62e8b, fb05d7f12b4241acb298cc040bbcc38f; no voters:
I20260620 07:20:36.982097 29800 raft_consensus.cc:2781] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:36.982328 29800 raft_consensus.cc:686] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [term 1 LEADER]: Becoming Leader. State: Replica: 7ed993b650504f72b32b94b632c62e8b, State: Running, Role: LEADER
I20260620 07:20:36.982648 29800 consensus_queue.cc:237] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 41457 } } peers { permanent_uuid: "fb05d7f12b4241acb298cc040bbcc38f" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42431 } } peers { permanent_uuid: "7ed993b650504f72b32b94b632c62e8b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34299 } }
I20260620 07:20:36.985745 29325 catalog_manager.cc:5495] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ed993b650504f72b32b94b632c62e8b (127.11.123.129). New cstate: current_term: 1 leader_uuid: "7ed993b650504f72b32b94b632c62e8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 41457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb05d7f12b4241acb298cc040bbcc38f" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ed993b650504f72b32b94b632c62e8b" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34299 } health_report { overall_health: HEALTHY } } }
W20260620 07:20:36.992321 29515 tablet.cc:2369] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:20:37.003979 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:37.000440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.11.123.129:34299)
0620 07:20:37.000549 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:37.000554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:37.000681 (+ 127us) server_negotiation.cc:407] Connection header received
0620 07:20:37.000855 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:37.000858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:37.000908 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:37.001012 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:20:37.001283 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:37.001783 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:37.002633 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:37.002848 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:37.002957 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:37.003252 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:20:37.003624 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:20:37.003754 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:37.003795 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 07:20:37.003843 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 07:20:37.009474 29599 raft_consensus.cc:1261] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f [term 1 FOLLOWER]: Refusing update from remote peer 7ed993b650504f72b32b94b632c62e8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:20:37.009474 29730 raft_consensus.cc:1261] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd [term 1 FOLLOWER]: Refusing update from remote peer 7ed993b650504f72b32b94b632c62e8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:20:37.010108 29803 consensus_queue.cc:1035] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f51ad2e7fa7748f1a01ab5830f3338dd" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 41457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:20:37.010313 29800 consensus_queue.cc:1035] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P 7ed993b650504f72b32b94b632c62e8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb05d7f12b4241acb298cc040bbcc38f" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:20:37.016709 29806 mvcc.cc:205] Tried to move back new op lower bound from 7298826391583371264 to 7298826391483408384. Current Snapshot: MvccSnapshot[applied={T|T < 7298826391583371264}]
I20260620 07:20:37.017524 29807 mvcc.cc:205] Tried to move back new op lower bound from 7298826391583371264 to 7298826391483408384. Current Snapshot: MvccSnapshot[applied={T|T < 7298826391583371264}]
I20260620 07:20:37.021266 29808 mvcc.cc:205] Tried to move back new op lower bound from 7298826391583371264 to 7298826391483408384. Current Snapshot: MvccSnapshot[applied={T|T < 7298826391583371264 or (T in {7298826391583371264})}]
I20260620 07:20:37.024942 29325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:20:37.025126 29325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:20:37.027408 29325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebe255045d6747db9cb351885a8c172a: Sending DeleteTablet for 3 replicas of tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51
I20260620 07:20:37.028098 29579 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:37 UTC) from {username='slave'} at 127.0.0.1:47868
I20260620 07:20:37.028308 29710 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:37 UTC) from {username='slave'} at 127.0.0.1:58642
I20260620 07:20:37.028532 29816 tablet_replica.cc:330] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f: stopping tablet replica
I20260620 07:20:37.028786 29817 tablet_replica.cc:330] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd: stopping tablet replica
I20260620 07:20:37.028765 29448 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:37 UTC) from {username='slave'} at 127.0.0.1:49082
I20260620 07:20:37.028836 29816 raft_consensus.cc:2229] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:20:37.029024 29817 raft_consensus.cc:2229] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:20:37.029129 29816 raft_consensus.cc:2258] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:20:37.029286 29817 raft_consensus.cc:2258] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:20:37.029629 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 29386
I20260620 07:20:37.031004 29816 ts_tablet_manager.cc:1905] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:20:37.034039 29816 ts_tablet_manager.cc:1918] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:20:37.034148 29816 log.cc:1196] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/6b3aa1eb88a745dc9da9b6b9c03dbe51
I20260620 07:20:37.034440 29816 ts_tablet_manager.cc:1939] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P fb05d7f12b4241acb298cc040bbcc38f: Deleting consensus metadata
I20260620 07:20:37.036206 29312 catalog_manager.cc:4841] TS fb05d7f12b4241acb298cc040bbcc38f (127.11.123.130:42431): tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51 (table test-table [id=ddd8558d830742d6b719562932cd3f99]) successfully deleted
I20260620 07:20:37.038117 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 29517
W20260620 07:20:37.039067 29313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:34299 (error 108)
I20260620 07:20:37.039986 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:37.039706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:34299 (local address 127.0.0.1:49104)
0620 07:20:37.039836 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:37.039906 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:34299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:20:37.040114 29313 catalog_manager.cc:4584] TS 7ed993b650504f72b32b94b632c62e8b (127.11.123.129:34299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51: Network error: Client connection negotiation failed: client connection to 127.11.123.129:34299: connect: Connection refused (error 111)
I20260620 07:20:37.040544 29817 ts_tablet_manager.cc:1905] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:20:37.042624 29817 ts_tablet_manager.cc:1918] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:20:37.042706 29817 log.cc:1196] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/6b3aa1eb88a745dc9da9b6b9c03dbe51
I20260620 07:20:37.043021 29817 ts_tablet_manager.cc:1939] T 6b3aa1eb88a745dc9da9b6b9c03dbe51 P f51ad2e7fa7748f1a01ab5830f3338dd: Deleting consensus metadata
I20260620 07:20:37.044086 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 29648
I20260620 07:20:37.044111 29313 catalog_manager.cc:4841] TS f51ad2e7fa7748f1a01ab5830f3338dd (127.11.123.131:41457): tablet 6b3aa1eb88a745dc9da9b6b9c03dbe51 (table test-table [id=ddd8558d830742d6b719562932cd3f99]) successfully deleted
I20260620 07:20:37.049724 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 29295
2026-06-20T07:20:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1002 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:37 dist-test-slave-j663 krb5kdc[29822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:37 dist-test-slave-j663 krb5kdc[29822](info): set up 2 sockets
Jun 20 07:20:37 dist-test-slave-j663 krb5kdc[29822](info): commencing operation
krb5kdc: starting...
W20260620 07:20:38.113054 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:38 dist-test-slave-j663 krb5kdc[29822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:20:38.238298 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:39803
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:39803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:38.351648 29835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:38.351933 29835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:38.352002 29835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:38.356175 29835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:38.356254 29835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:38.356300 29835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:38.356357 29835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:38.356395 29835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:38.361662 29835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:39803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:39803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:38.362694 29835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:38.363708 29835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:38.372663 29841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:38.372629 29840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:38.372656 29843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:38.372961 29835 server_base.cc:1034] running on GCE node
I20260620 07:20:38.373826 29835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:38.375393 29835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:38.376633 29835 hybrid_clock.cc:648] HybridClock initialized: now 1781940038376595 us; error 50 us; skew 500 ppm
Jun 20 07:20:38 dist-test-slave-j663 krb5kdc[29822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940038, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:38.379603 29835 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:20:38.380466 29835 webserver.cc:460] Webserver started at http://127.11.123.190:32877/ using document root <none> and password file <none>
I20260620 07:20:38.381021 29835 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:38.381162 29835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:38.381402 29835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:38.383423 29835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "24480a61164a4346af0771b7f005b3a9"
format_stamp: "Formatted at 2026-06-20 07:20:38 on dist-test-slave-j663"
server_key: "ca8a19b2c163bfa5ff096212e8d90420"
server_key_iv: "9cb47ea550b981ac87d7e117b1eefc43"
server_key_version: "encryptionkey@0"
I20260620 07:20:38.383977 29835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "24480a61164a4346af0771b7f005b3a9"
format_stamp: "Formatted at 2026-06-20 07:20:38 on dist-test-slave-j663"
server_key: "ca8a19b2c163bfa5ff096212e8d90420"
server_key_iv: "9cb47ea550b981ac87d7e117b1eefc43"
server_key_version: "encryptionkey@0"
I20260620 07:20:38.387732 29835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:20:38.390112 29850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:38.391302 29835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:20:38.391428 29835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "24480a61164a4346af0771b7f005b3a9"
format_stamp: "Formatted at 2026-06-20 07:20:38 on dist-test-slave-j663"
server_key: "ca8a19b2c163bfa5ff096212e8d90420"
server_key_iv: "9cb47ea550b981ac87d7e117b1eefc43"
server_key_version: "encryptionkey@0"
I20260620 07:20:38.391535 29835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:38.407250 29835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:38.409968 29835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:38.410172 29835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:38.417004 29835 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:39803
I20260620 07:20:38.417069 29902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:39803 every 8 connection(s)
I20260620 07:20:38.417999 29835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:38.421061 29903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:38.425010 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 29835
I20260620 07:20:38.425205 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:38.425514 11758 external_mini_cluster.cc:1428] Setting key e0a03398eb49958fd5234838c2f32e0a
I20260620 07:20:38.426913 29903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9: Bootstrap starting.
I20260620 07:20:38.429564 29903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:38.430438 29903 log.cc:824] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:38.432435 29903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9: No bootstrap required, opened a new log
Jun 20 07:20:38 dist-test-slave-j663 krb5kdc[29822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:20:38.436739 29903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24480a61164a4346af0771b7f005b3a9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39803 } }
I20260620 07:20:38.436950 29903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:38.437012 29903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24480a61164a4346af0771b7f005b3a9, State: Initialized, Role: FOLLOWER
I20260620 07:20:38.437561 29903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24480a61164a4346af0771b7f005b3a9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39803 } }
I20260620 07:20:38.437736 29903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:38.437796 29903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:38.437908 29903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:38.438925 29903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24480a61164a4346af0771b7f005b3a9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39803 } }
I20260620 07:20:38.439311 29903 leader_election.cc:302] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24480a61164a4346af0771b7f005b3a9; no voters:
I20260620 07:20:38.439677 29903 leader_election.cc:288] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:38.439826 29908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:38.440059 29908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [term 1 LEADER]: Becoming Leader. State: Replica: 24480a61164a4346af0771b7f005b3a9, State: Running, Role: LEADER
I20260620 07:20:38.440369 29908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24480a61164a4346af0771b7f005b3a9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39803 } }
I20260620 07:20:38.440850 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:38.426838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.11.123.190:39803)
0620 07:20:38.427260 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:38.427278 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:38.427319 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:20:38.428149 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:38.428172 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:38.428573 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:38.428922 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:38.429683 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:38.431074 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:38.431940 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:38.432587 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:38.432758 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:38.435584 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:38.435590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:38.435602 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:38.435628 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:38.437397 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:38.437918 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:38.437924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:38.437930 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:38.438103 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:38.438452 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:38.438455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:38.438456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:38.438917 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:38.439161 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:38.439391 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 07:20:38.439808 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":133,"threads_started":1}
I20260620 07:20:38.440879 29903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:38.442081 29909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24480a61164a4346af0771b7f005b3a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24480a61164a4346af0771b7f005b3a9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39803 } } }
I20260620 07:20:38.442212 29909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:38.442452 29909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24480a61164a4346af0771b7f005b3a9. Latest consensus state: current_term: 1 leader_uuid: "24480a61164a4346af0771b7f005b3a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24480a61164a4346af0771b7f005b3a9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 39803 } } }
I20260620 07:20:38.442543 29909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:38.443614 29917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:38.446496 29917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:38.451725 29917 catalog_manager.cc:1347] Generated new cluster ID: 651be86232f74875bc8142460d2f38b9
I20260620 07:20:38.451792 29917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:38.482563 29917 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:38.483873 29917 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:38.494079 29917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9: Generated new TSK 0
I20260620 07:20:38.494891 29917 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:20:38.524484 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39803
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:38.647855 29929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:38.648137 29929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:38.648237 29929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:38.652223 29929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:38.652300 29929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:38.652395 29929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:20:38.657723 29929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:38.658885 29929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:38.659927 29929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:38.669144 29934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:38.669211 29929 server_base.cc:1034] running on GCE node
W20260620 07:20:38.669121 29935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:38.669117 29937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:38.669808 29929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:38.670392 29929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:38.671550 29929 hybrid_clock.cc:648] HybridClock initialized: now 1781940038671526 us; error 35 us; skew 500 ppm
Jun 20 07:20:38 dist-test-slave-j663 krb5kdc[29822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940038, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:38.674482 29929 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:20:38.675345 29929 webserver.cc:460] Webserver started at http://127.11.123.129:35235/ using document root <none> and password file <none>
I20260620 07:20:38.675889 29929 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:38.675977 29929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:38.676209 29929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:38.678231 29929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "353221dbe22d4ce1b6612c39fe5878f9"
format_stamp: "Formatted at 2026-06-20 07:20:38 on dist-test-slave-j663"
server_key: "5eceff0c2fe0622b921f80accf3036ae"
server_key_iv: "2f19e1f277d8b58890398ab2faa083f8"
server_key_version: "encryptionkey@0"
I20260620 07:20:38.678850 29929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "353221dbe22d4ce1b6612c39fe5878f9"
format_stamp: "Formatted at 2026-06-20 07:20:38 on dist-test-slave-j663"
server_key: "5eceff0c2fe0622b921f80accf3036ae"
server_key_iv: "2f19e1f277d8b58890398ab2faa083f8"
server_key_version: "encryptionkey@0"
I20260620 07:20:38.682312 29929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:20:38.684564 29944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:38.685767 29929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:20:38.685863 29929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "353221dbe22d4ce1b6612c39fe5878f9"
format_stamp: "Formatted at 2026-06-20 07:20:38 on dist-test-slave-j663"
server_key: "5eceff0c2fe0622b921f80accf3036ae"
server_key_iv: "2f19e1f277d8b58890398ab2faa083f8"
server_key_version: "encryptionkey@0"
I20260620 07:20:38.685941 29929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:38.705624 29929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:38.714550 29929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:38.714752 29929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:38.715283 29929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:38.716069 29929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:38.716113 29929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:38.716161 29929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:38.716181 29929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:38.725682 30057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:43139 every 8 connection(s)
I20260620 07:20:38.725678 29929 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:43139
I20260620 07:20:38.726662 29929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:20:38.731199 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 29929
I20260620 07:20:38.731346 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:20:38.731640 11758 external_mini_cluster.cc:1428] Setting key 74e4d52605ca4801b835aa86e51a1c84
Jun 20 07:20:38 dist-test-slave-j663 krb5kdc[29822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940038, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:20:38.740701 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:38.728534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:59505 (local address 127.11.123.190:39803)
0620 07:20:38.728679 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:38.728683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:38.729517 (+ 834us) server_negotiation.cc:407] Connection header received
0620 07:20:38.730817 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:38.730820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:38.730869 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:38.730963 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:38.732193 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:38.732729 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:38.733941 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:38.734350 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:38.734472 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:38.737240 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:38.737243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:38.737246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:38.737281 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:38.738701 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:38.739202 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:38.739206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:38.739207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:38.739292 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:38.739651 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:38.739654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:38.739656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:38.739838 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:38.739946 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:38.740418 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 07:20:38.740540 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:20:38.741673 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:38.728790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:39803 (local address 127.11.123.129:59505)
0620 07:20:38.729371 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:38.729413 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:38.730625 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:38.731066 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:38.731072 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:38.731507 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:38.732025 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:38.732038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:38.732864 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:38.732866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:38.733816 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:38.733825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:38.734526 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:38.734532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:38.734804 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:38.735417 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:20:38.735438 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:38.737010 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:38.738820 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:38.738826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:38.738839 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:38.739090 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:38.739393 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:38.739396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:38.739397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:38.739546 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:38.739945 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:38.739951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:38.740185 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 07:20:38.740362 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 07:20:38.740704 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":104,"threads_started":1}
I20260620 07:20:38.743181 30058 heartbeater.cc:347] Connected to a master server at 127.11.123.190:39803
I20260620 07:20:38.743453 30058 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
I20260620 07:20:38.743990 30058 heartbeater.cc:511] Master 127.11.123.190:39803 requested a full tablet report, sending...
I20260620 07:20:38.745524 29867 ts_manager.cc:194] Registered new tserver with Master: 353221dbe22d4ce1b6612c39fe5878f9 (127.11.123.129:43139)
I20260620 07:20:38.746630 29867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:59505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:20:38.759753 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39803
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:38.872138 30063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:38.872397 30063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:38.872440 30063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:38.876504 30063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:38.876564 30063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:38.876621 30063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:20:38.881888 30063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:38.882828 30063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:38.883760 30063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:38.894068 30068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:38.894066 30069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:38.894134 30071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:38.894331 30063 server_base.cc:1034] running on GCE node
I20260620 07:20:38.894838 30063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:38.895500 30063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:38.896723 30063 hybrid_clock.cc:648] HybridClock initialized: now 1781940038896711 us; error 39 us; skew 500 ppm
Jun 20 07:20:38 dist-test-slave-j663 krb5kdc[29822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940038, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:38.899767 30063 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:20:38.900601 30063 webserver.cc:460] Webserver started at http://127.11.123.130:38105/ using document root <none> and password file <none>
I20260620 07:20:38.901250 30063 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:38.901346 30063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:38.901582 30063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:38.903585 30063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "69ab9a8abd1c47ca9baea4806ada4ab7"
format_stamp: "Formatted at 2026-06-20 07:20:38 on dist-test-slave-j663"
server_key: "09ebae84d00fdbcf295ce9038f30b8d8"
server_key_iv: "2d822c890ccc3843ab7e2ecd7932d1c4"
server_key_version: "encryptionkey@0"
I20260620 07:20:38.904160 30063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "69ab9a8abd1c47ca9baea4806ada4ab7"
format_stamp: "Formatted at 2026-06-20 07:20:38 on dist-test-slave-j663"
server_key: "09ebae84d00fdbcf295ce9038f30b8d8"
server_key_iv: "2d822c890ccc3843ab7e2ecd7932d1c4"
server_key_version: "encryptionkey@0"
I20260620 07:20:38.908248 30063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:20:38.910657 30078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:38.911867 30063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:20:38.912003 30063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "69ab9a8abd1c47ca9baea4806ada4ab7"
format_stamp: "Formatted at 2026-06-20 07:20:38 on dist-test-slave-j663"
server_key: "09ebae84d00fdbcf295ce9038f30b8d8"
server_key_iv: "2d822c890ccc3843ab7e2ecd7932d1c4"
server_key_version: "encryptionkey@0"
I20260620 07:20:38.912111 30063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:38.946851 30063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:38.950049 30063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:38.950254 30063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:38.950861 30063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:38.951683 30063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:38.951754 30063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:38.951825 30063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:38.951877 30063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:38.961074 30063 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:42373
I20260620 07:20:38.961136 30191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:42373 every 8 connection(s)
I20260620 07:20:38.961954 30063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:20:38.966583 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 30063
I20260620 07:20:38.966715 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:20:38.967016 11758 external_mini_cluster.cc:1428] Setting key 23c184aefa25f1e50376c329a51a92f2
Jun 20 07:20:38 dist-test-slave-j663 krb5kdc[29822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940038, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
I20260620 07:20:38.975744 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:38.963727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:37295 (local address 127.11.123.190:39803)
0620 07:20:38.963874 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:38.963878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:38.964650 (+ 772us) server_negotiation.cc:407] Connection header received
0620 07:20:38.966092 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:38.966096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:38.966152 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:38.966274 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:38.967556 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:38.968282 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:38.969485 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:38.969717 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:38.969827 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:38.972238 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:38.972242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:38.972245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:38.972271 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:38.973752 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:38.974248 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:38.974253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:38.974255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:38.974338 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:38.974677 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:38.974683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:38.974688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:38.974870 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:38.974980 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:38.975454 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 07:20:38.975574 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:20:38.976517 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:38.964056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:39803 (local address 127.11.123.130:37295)
0620 07:20:38.964499 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:38.964540 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:38.965880 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:38.966447 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:38.966453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:38.966901 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:38.967408 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:38.967423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:38.968412 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:38.968416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:38.969355 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:38.969364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:38.969865 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:38.969869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:38.970043 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:38.970650 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:20:38.970666 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:38.972074 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:38.973873 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:38.973879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:38.973893 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:38.974144 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:38.974438 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:38.974443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:38.974445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:38.974583 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:38.974978 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:38.974985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:38.975227 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 07:20:38.975413 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 07:20:38.975733 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":99,"threads_started":1}
I20260620 07:20:38.977788 30192 heartbeater.cc:347] Connected to a master server at 127.11.123.190:39803
I20260620 07:20:38.978032 30192 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
I20260620 07:20:38.978547 30192 heartbeater.cc:511] Master 127.11.123.190:39803 requested a full tablet report, sending...
I20260620 07:20:38.979661 29867 ts_manager.cc:194] Registered new tserver with Master: 69ab9a8abd1c47ca9baea4806ada4ab7 (127.11.123.130:42373)
I20260620 07:20:38.980517 29867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:37295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:20:38.991881 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39803
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:39.102842 30197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:39.103096 30197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:39.103164 30197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:39.107265 30197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:39.107316 30197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:39.107399 30197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:20:39.112633 30197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:39803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:39.113668 30197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:39.114656 30197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:39.124817 30203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:39.124837 30202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:39.124837 30205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:39.125104 30197 server_base.cc:1034] running on GCE node
I20260620 07:20:39.125540 30197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:39.126312 30197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:39.133728 30197 hybrid_clock.cc:648] HybridClock initialized: now 1781940039133704 us; error 40 us; skew 500 ppm
Jun 20 07:20:39 dist-test-slave-j663 krb5kdc[29822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940039, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:39.136502 30197 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:20:39.137483 30197 webserver.cc:460] Webserver started at http://127.11.123.131:36115/ using document root <none> and password file <none>
I20260620 07:20:39.138058 30197 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:39.138152 30197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:39.138386 30197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:39.140817 30197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "e4de62f38e0c4b3db2734de16cd7cdba"
format_stamp: "Formatted at 2026-06-20 07:20:39 on dist-test-slave-j663"
server_key: "cedad673973afddaf0d6d2a2cfcdb54b"
server_key_iv: "ee6067c54990c3ddf22d570e7757c31f"
server_key_version: "encryptionkey@0"
I20260620 07:20:39.141484 30197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "e4de62f38e0c4b3db2734de16cd7cdba"
format_stamp: "Formatted at 2026-06-20 07:20:39 on dist-test-slave-j663"
server_key: "cedad673973afddaf0d6d2a2cfcdb54b"
server_key_iv: "ee6067c54990c3ddf22d570e7757c31f"
server_key_version: "encryptionkey@0"
I20260620 07:20:39.145287 30197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:20:39.147622 30212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:39.148859 30197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:20:39.148994 30197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "e4de62f38e0c4b3db2734de16cd7cdba"
format_stamp: "Formatted at 2026-06-20 07:20:39 on dist-test-slave-j663"
server_key: "cedad673973afddaf0d6d2a2cfcdb54b"
server_key_iv: "ee6067c54990c3ddf22d570e7757c31f"
server_key_version: "encryptionkey@0"
I20260620 07:20:39.149148 30197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:39.167246 30197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:39.170296 30197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:39.170495 30197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:39.171075 30197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:39.171882 30197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:39.171954 30197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:39.172027 30197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:39.172073 30197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:39.181155 30197 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:45661
I20260620 07:20:39.181180 30325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:45661 every 8 connection(s)
I20260620 07:20:39.182011 30197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:20:39.188138 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 30197
I20260620 07:20:39.188297 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:20:39.188622 11758 external_mini_cluster.cc:1428] Setting key e4f0fc59bd10d7f0dafcf888e5e79f61
Jun 20 07:20:39 dist-test-slave-j663 krb5kdc[29822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940039, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:20:39.195441 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.183682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:58517 (local address 127.11.123.190:39803)
0620 07:20:39.183839 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:39.183843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:39.184653 (+ 810us) server_negotiation.cc:407] Connection header received
0620 07:20:39.186010 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:39.186013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:39.186061 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:39.186141 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:39.187187 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.187696 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.189310 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.189654 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.189749 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.192373 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:39.192376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:39.192378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:39.192401 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:39.193729 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:39.194177 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:39.194180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:39.194181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:39.194252 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:39.194545 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:39.194547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:39.194548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:39.194723 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:39.194799 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:39.195237 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 07:20:39.195329 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:20:39.196319 30328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.184079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:39803 (local address 127.11.123.131:58517)
0620 07:20:39.184498 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:39.184540 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:39.185819 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:39.186261 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:39.186267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:39.186592 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:39.187058 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.187070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.187803 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.187805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.189160 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.189170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.189820 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.189824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.190006 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.190663 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:20:39.190717 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:39.192220 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:39.193831 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:39.193835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:39.193846 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:39.194085 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:39.194332 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:39.194334 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:39.194335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:39.194467 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:39.194803 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:39.194808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:39.195038 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 07:20:39.195196 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 07:20:39.195424 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20260620 07:20:39.197434 30326 heartbeater.cc:347] Connected to a master server at 127.11.123.190:39803
I20260620 07:20:39.197661 30326 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:39.198146 30326 heartbeater.cc:511] Master 127.11.123.190:39803 requested a full tablet report, sending...
I20260620 07:20:39.199146 29867 ts_manager.cc:194] Registered new tserver with Master: e4de62f38e0c4b3db2734de16cd7cdba (127.11.123.131:45661)
I20260620 07:20:39.199878 29867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:58517
I20260620 07:20:39.202391 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:20:39.212033 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.204241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.11.123.190:39803)
0620 07:20:39.204383 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:39.204388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:39.204588 (+ 200us) server_negotiation.cc:407] Connection header received
0620 07:20:39.204741 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:39.204744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:39.204787 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:39.204874 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:39.205625 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.206134 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.206987 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.207221 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.207331 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.208916 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:39.208920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:39.208923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:39.208947 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:39.210197 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:39.210711 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:39.210714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:39.210716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:39.210798 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:39.211129 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:39.211132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:39.211134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:39.211332 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:39.211494 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:39.211775 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 07:20:39.211867 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:20:39.214877 29867 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:20:39.217960 29867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:20:39.233319 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.227204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.11.123.130:42373)
0620 07:20:39.227634 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:39.227647 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:39.227672 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:20:39.227747 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:39.227753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:39.227956 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:39.229008 (+ 1052us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:39.229214 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.230369 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.231979 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.232606 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.232758 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.232891 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:39.233042 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 07:20:39.233166 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":127,"threads_started":1}
I20260620 07:20:39.233372 30338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.226373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:45661 (local address 127.0.0.1:36092)
0620 07:20:39.227464 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:39.227479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:39.227570 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:39.229217 (+ 1647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:39.229220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:39.229239 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:39.229329 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.229335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.230762 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.230767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.231752 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.231760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.232913 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.232917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.233015 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.233039 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:20:39.233127 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 07:20:39.233200 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260620 07:20:39.233338 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.226249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:42373 (local address 127.0.0.1:40914)
0620 07:20:39.226875 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:39.226928 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:39.227058 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:39.228676 (+ 1618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:39.228682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:39.228707 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:39.228854 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.228861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.230499 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.230502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.231844 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.231853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.232743 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.232746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.232850 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.232886 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:20:39.233002 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 07:20:39.233150 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":73,"thread_start_us":90,"threads_started":1}
I20260620 07:20:39.234014 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.226639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.11.123.131:45661)
0620 07:20:39.228735 (+ 2096us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:39.228746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:39.228766 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:20:39.228837 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:39.228841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:39.229018 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:39.229471 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:39.229594 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.230636 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.231919 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.232766 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.233577 (+ 811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.233733 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:39.233779 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 07:20:39.233865 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2001,"thread_start_us":104,"threads_started":1}
I20260620 07:20:39.234328 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.227735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.11.123.129:43139)
0620 07:20:39.228021 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:39.228036 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:39.228302 (+ 266us) server_negotiation.cc:407] Connection header received
0620 07:20:39.228421 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:39.228457 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:39.228660 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:39.229062 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:39.229486 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.230214 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.232135 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.232943 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.233049 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.233219 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:39.234080 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 07:20:39.234185 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":123,"threads_started":1}
I20260620 07:20:39.234539 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.227319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:43139 (local address 127.0.0.1:47190)
0620 07:20:39.228178 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:39.228191 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:39.228291 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:39.229243 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:39.229246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:39.229279 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:39.229369 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.229373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.230343 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.230348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.232023 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.232033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.233282 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.233286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.233375 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.233945 (+ 570us) client_negotiation.cc:769] Sending connection context
0620 07:20:39.234034 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 07:20:39.234285 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":47,"spinlock_wait_cycles":3328,"thread_start_us":93,"threads_started":1}
I20260620 07:20:39.236582 30260 tablet_service.cc:1467] Processing CreateTablet for tablet 97dc053033314db0b2e14faa418ecd9c (DEFAULT_TABLE table=test-table [id=a60d627dd1a5451ab4667ede7c6889ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:39.236584 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 97dc053033314db0b2e14faa418ecd9c (DEFAULT_TABLE table=test-table [id=a60d627dd1a5451ab4667ede7c6889ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:39.237787 30260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dc053033314db0b2e14faa418ecd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:39.238319 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dc053033314db0b2e14faa418ecd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:39.239370 29992 tablet_service.cc:1467] Processing CreateTablet for tablet 97dc053033314db0b2e14faa418ecd9c (DEFAULT_TABLE table=test-table [id=a60d627dd1a5451ab4667ede7c6889ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:39.240200 29992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dc053033314db0b2e14faa418ecd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:39.246133 30343 tablet_bootstrap.cc:492] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7: Bootstrap starting.
I20260620 07:20:39.246419 30345 tablet_bootstrap.cc:492] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9: Bootstrap starting.
I20260620 07:20:39.246650 30344 tablet_bootstrap.cc:492] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba: Bootstrap starting.
I20260620 07:20:39.248553 30345 tablet_bootstrap.cc:654] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:39.248553 30343 tablet_bootstrap.cc:654] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:39.248857 30344 tablet_bootstrap.cc:654] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:39.249528 30345 log.cc:824] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:39.249852 30343 log.cc:824] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:39.250036 30344 log.cc:824] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:39.251657 30345 tablet_bootstrap.cc:492] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9: No bootstrap required, opened a new log
I20260620 07:20:39.251833 30344 tablet_bootstrap.cc:492] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba: No bootstrap required, opened a new log
I20260620 07:20:39.251833 30343 tablet_bootstrap.cc:492] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7: No bootstrap required, opened a new log
I20260620 07:20:39.251860 30345 ts_tablet_manager.cc:1397] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:20:39.252012 30344 ts_tablet_manager.cc:1397] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 07:20:39.252012 30343 ts_tablet_manager.cc:1397] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:20:39.255337 30345 raft_consensus.cc:348] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42373 } } peers { permanent_uuid: "e4de62f38e0c4b3db2734de16cd7cdba" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 45661 } } peers { permanent_uuid: "353221dbe22d4ce1b6612c39fe5878f9" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 43139 } }
I20260620 07:20:39.255585 30345 raft_consensus.cc:374] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:39.255640 30345 raft_consensus.cc:729] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353221dbe22d4ce1b6612c39fe5878f9, State: Initialized, Role: FOLLOWER
I20260620 07:20:39.255806 30343 raft_consensus.cc:348] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42373 } } peers { permanent_uuid: "e4de62f38e0c4b3db2734de16cd7cdba" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 45661 } } peers { permanent_uuid: "353221dbe22d4ce1b6612c39fe5878f9" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 43139 } }
I20260620 07:20:39.255806 30344 raft_consensus.cc:348] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42373 } } peers { permanent_uuid: "e4de62f38e0c4b3db2734de16cd7cdba" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 45661 } } peers { permanent_uuid: "353221dbe22d4ce1b6612c39fe5878f9" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 43139 } }
I20260620 07:20:39.256055 30343 raft_consensus.cc:374] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:39.256055 30344 raft_consensus.cc:374] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:39.256114 30344 raft_consensus.cc:729] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4de62f38e0c4b3db2734de16cd7cdba, State: Initialized, Role: FOLLOWER
I20260620 07:20:39.256122 30343 raft_consensus.cc:729] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab9a8abd1c47ca9baea4806ada4ab7, State: Initialized, Role: FOLLOWER
I20260620 07:20:39.256109 30345 consensus_queue.cc:260] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42373 } } peers { permanent_uuid: "e4de62f38e0c4b3db2734de16cd7cdba" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 45661 } } peers { permanent_uuid: "353221dbe22d4ce1b6612c39fe5878f9" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 43139 } }
I20260620 07:20:39.256567 30343 consensus_queue.cc:260] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42373 } } peers { permanent_uuid: "e4de62f38e0c4b3db2734de16cd7cdba" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 45661 } } peers { permanent_uuid: "353221dbe22d4ce1b6612c39fe5878f9" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 43139 } }
I20260620 07:20:39.256814 30058 heartbeater.cc:503] Master 127.11.123.190:39803 was elected leader, sending a full tablet report...
I20260620 07:20:39.257265 30192 heartbeater.cc:503] Master 127.11.123.190:39803 was elected leader, sending a full tablet report...
I20260620 07:20:39.257501 30343 ts_tablet_manager.cc:1428] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 07:20:39.256567 30344 consensus_queue.cc:260] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42373 } } peers { permanent_uuid: "e4de62f38e0c4b3db2734de16cd7cdba" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 45661 } } peers { permanent_uuid: "353221dbe22d4ce1b6612c39fe5878f9" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 43139 } }
I20260620 07:20:39.258296 30345 ts_tablet_manager.cc:1428] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 07:20:39.259352 30326 heartbeater.cc:503] Master 127.11.123.190:39803 was elected leader, sending a full tablet report...
I20260620 07:20:39.259625 30344 ts_tablet_manager.cc:1428] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20260620 07:20:39.432330 30350 raft_consensus.cc:482] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:20:39.432579 30350 raft_consensus.cc:504] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42373 } } peers { permanent_uuid: "e4de62f38e0c4b3db2734de16cd7cdba" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 45661 } } peers { permanent_uuid: "353221dbe22d4ce1b6612c39fe5878f9" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 43139 } }
W20260620 07:20:39.433041 30327 tablet.cc:2369] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:20:39.433965 30350 leader_election.cc:288] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4de62f38e0c4b3db2734de16cd7cdba (127.11.123.131:45661), 353221dbe22d4ce1b6612c39fe5878f9 (127.11.123.129:43139)
I20260620 07:20:39.439226 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.434270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:33613 (local address 127.11.123.131:45661)
0620 07:20:39.434396 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:39.434401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:39.434577 (+ 176us) server_negotiation.cc:407] Connection header received
0620 07:20:39.434753 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:39.434761 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:39.434822 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:39.434910 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:39.435278 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.436075 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.437939 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.438546 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.438672 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.438725 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:39.439022 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 07:20:39.439095 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 07:20:39.439226 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.434427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:46471 (local address 127.11.123.129:43139)
0620 07:20:39.434590 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:39.434595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:39.434764 (+ 169us) server_negotiation.cc:407] Connection header received
0620 07:20:39.434824 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:39.434827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:39.434884 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:39.434963 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:39.435278 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.436075 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.437939 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.438547 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.438673 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.438725 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:39.439022 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 07:20:39.439095 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:20:39.439263 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.434157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:45661 (local address 127.11.123.130:33613)
0620 07:20:39.434494 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:39.434510 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:39.434618 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:39.434956 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:39.434961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:39.434980 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:39.435142 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.435149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.436214 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.436218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.437808 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.437816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.438673 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.438677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.438819 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.438911 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 07:20:39.438992 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 07:20:39.439103 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":236,"spinlock_wait_cycles":3840}
I20260620 07:20:39.439275 30352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.434236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:43139 (local address 127.11.123.130:46471)
0620 07:20:39.434633 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:39.434647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:39.434761 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:39.434982 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:39.434985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:39.434995 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:39.435144 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.435149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.436214 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.436218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.437805 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.437816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.438673 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.438700 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.438836 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.438913 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 07:20:39.438992 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 07:20:39.439103 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":237,"thread_start_us":94,"threads_started":1}
I20260620 07:20:39.439996 30012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dc053033314db0b2e14faa418ecd9c" candidate_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353221dbe22d4ce1b6612c39fe5878f9" is_pre_election: true
I20260620 07:20:39.440016 30280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dc053033314db0b2e14faa418ecd9c" candidate_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4de62f38e0c4b3db2734de16cd7cdba" is_pre_election: true
I20260620 07:20:39.440304 30012 raft_consensus.cc:2454] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab9a8abd1c47ca9baea4806ada4ab7 in term 0.
I20260620 07:20:39.440296 30280 raft_consensus.cc:2454] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab9a8abd1c47ca9baea4806ada4ab7 in term 0.
I20260620 07:20:39.440785 30081 leader_election.cc:302] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353221dbe22d4ce1b6612c39fe5878f9, 69ab9a8abd1c47ca9baea4806ada4ab7; no voters:
I20260620 07:20:39.441057 30350 raft_consensus.cc:2781] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:20:39.441201 30350 raft_consensus.cc:482] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:20:39.441257 30350 raft_consensus.cc:3037] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:39.442353 30350 raft_consensus.cc:504] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42373 } } peers { permanent_uuid: "e4de62f38e0c4b3db2734de16cd7cdba" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 45661 } } peers { permanent_uuid: "353221dbe22d4ce1b6612c39fe5878f9" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 43139 } }
I20260620 07:20:39.442749 30350 leader_election.cc:288] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [CANDIDATE]: Term 1 election: Requested vote from peers e4de62f38e0c4b3db2734de16cd7cdba (127.11.123.131:45661), 353221dbe22d4ce1b6612c39fe5878f9 (127.11.123.129:43139)
I20260620 07:20:39.443063 30280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dc053033314db0b2e14faa418ecd9c" candidate_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4de62f38e0c4b3db2734de16cd7cdba"
I20260620 07:20:39.443135 30012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dc053033314db0b2e14faa418ecd9c" candidate_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353221dbe22d4ce1b6612c39fe5878f9"
I20260620 07:20:39.443187 30280 raft_consensus.cc:3037] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:39.443248 30012 raft_consensus.cc:3037] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:39.444509 30280 raft_consensus.cc:2454] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab9a8abd1c47ca9baea4806ada4ab7 in term 1.
I20260620 07:20:39.444510 30012 raft_consensus.cc:2454] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab9a8abd1c47ca9baea4806ada4ab7 in term 1.
I20260620 07:20:39.444825 30081 leader_election.cc:302] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353221dbe22d4ce1b6612c39fe5878f9, 69ab9a8abd1c47ca9baea4806ada4ab7; no voters:
I20260620 07:20:39.444993 30350 raft_consensus.cc:2781] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:39.445256 30350 raft_consensus.cc:686] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab9a8abd1c47ca9baea4806ada4ab7, State: Running, Role: LEADER
I20260620 07:20:39.445570 30350 consensus_queue.cc:237] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42373 } } peers { permanent_uuid: "e4de62f38e0c4b3db2734de16cd7cdba" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 45661 } } peers { permanent_uuid: "353221dbe22d4ce1b6612c39fe5878f9" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 43139 } }
I20260620 07:20:39.448945 29865 catalog_manager.cc:5495] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab9a8abd1c47ca9baea4806ada4ab7 (127.11.123.130). New cstate: current_term: 1 leader_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab9a8abd1c47ca9baea4806ada4ab7" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 42373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4de62f38e0c4b3db2734de16cd7cdba" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 45661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353221dbe22d4ce1b6612c39fe5878f9" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 43139 } health_report { overall_health: UNKNOWN } } }
W20260620 07:20:39.463002 30193 tablet.cc:2369] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:20:39.471022 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.466895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.11.123.130:42373)
0620 07:20:39.467032 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:39.467039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:39.467057 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:20:39.467140 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:39.467146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:39.467227 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:39.467357 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:20:39.467684 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.468517 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.469455 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.469703 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.469787 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.470132 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:20:39.470516 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:20:39.470624 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:39.470798 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 07:20:39.470859 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:20:39.476745 30280 raft_consensus.cc:1261] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba [term 1 FOLLOWER]: Refusing update from remote peer 69ab9a8abd1c47ca9baea4806ada4ab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:20:39.476745 30012 raft_consensus.cc:1261] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9 [term 1 FOLLOWER]: Refusing update from remote peer 69ab9a8abd1c47ca9baea4806ada4ab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:20:39.477465 30353 consensus_queue.cc:1035] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "353221dbe22d4ce1b6612c39fe5878f9" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 43139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:20:39.477687 30350 consensus_queue.cc:1035] T 97dc053033314db0b2e14faa418ecd9c P 69ab9a8abd1c47ca9baea4806ada4ab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4de62f38e0c4b3db2734de16cd7cdba" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 45661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:20:39.477954 30059 tablet.cc:2369] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:20:39.484148 30356 mvcc.cc:205] Tried to move back new op lower bound from 7298826401689706496 to 7298826401571778560. Current Snapshot: MvccSnapshot[applied={T|T < 7298826401689706496}]
I20260620 07:20:39.484304 30358 mvcc.cc:205] Tried to move back new op lower bound from 7298826401689706496 to 7298826401571778560. Current Snapshot: MvccSnapshot[applied={T|T < 7298826401689706496}]
I20260620 07:20:39.493788 29865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:20:39.493965 29865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:20:39.496356 29865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24480a61164a4346af0771b7f005b3a9: Sending DeleteTablet for 3 replicas of tablet 97dc053033314db0b2e14faa418ecd9c
I20260620 07:20:39.497162 30126 tablet_service.cc:1514] Processing DeleteTablet for tablet 97dc053033314db0b2e14faa418ecd9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:39 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:40914
I20260620 07:20:39.497203 29992 tablet_service.cc:1514] Processing DeleteTablet for tablet 97dc053033314db0b2e14faa418ecd9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:39 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:47190
I20260620 07:20:39.497265 30260 tablet_service.cc:1514] Processing DeleteTablet for tablet 97dc053033314db0b2e14faa418ecd9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:39 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:36092
I20260620 07:20:39.497684 30368 tablet_replica.cc:330] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba: stopping tablet replica
I20260620 07:20:39.497958 30367 tablet_replica.cc:330] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9: stopping tablet replica
I20260620 07:20:39.498016 30368 raft_consensus.cc:2229] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:20:39.498232 30367 raft_consensus.cc:2229] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:20:39.498323 30368 raft_consensus.cc:2258] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:20:39.498461 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 29929
I20260620 07:20:39.498499 30367 raft_consensus.cc:2258] T 97dc053033314db0b2e14faa418ecd9c P 353221dbe22d4ce1b6612c39fe5878f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:20:39.500564 30368 ts_tablet_manager.cc:1905] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:20:39.503400 30368 ts_tablet_manager.cc:1918] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:20:39.503500 30368 log.cc:1196] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/97dc053033314db0b2e14faa418ecd9c
I20260620 07:20:39.503846 30368 ts_tablet_manager.cc:1939] T 97dc053033314db0b2e14faa418ecd9c P e4de62f38e0c4b3db2734de16cd7cdba: Deleting consensus metadata
I20260620 07:20:39.505054 29854 catalog_manager.cc:4841] TS e4de62f38e0c4b3db2734de16cd7cdba (127.11.123.131:45661): tablet 97dc053033314db0b2e14faa418ecd9c (table test-table [id=a60d627dd1a5451ab4667ede7c6889ff]) successfully deleted
W20260620 07:20:39.506973 29853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:43139 (error 108)
I20260620 07:20:39.507118 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 30063
I20260620 07:20:39.509811 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.509506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:43139 (local address 127.0.0.1:47200)
0620 07:20:39.509672 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:39.509740 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:43139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:20:39.510066 29853 catalog_manager.cc:4584] TS 353221dbe22d4ce1b6612c39fe5878f9 (127.11.123.129:43139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97dc053033314db0b2e14faa418ecd9c: Network error: Client connection negotiation failed: client connection to 127.11.123.129:43139: connect: Connection refused (error 111)
I20260620 07:20:39.513413 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 30197
I20260620 07:20:39.515177 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.513526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:42373 (local address 127.0.0.1:40926)
0620 07:20:39.515055 (+ 1529us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:39.515109 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.130:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1438}
I20260620 07:20:39.518568 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 29835
2026-06-20T07:20:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2469 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T07:20:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:39Z Disabled control of system clock
I20260620 07:20:39.548998 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:34891
--webserver_interface=127.11.123.190
--webserver_port=0
--builtin_ntp_servers=127.11.123.148:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:34891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:20:39.656083 30373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:39.656349 30373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:39.656385 30373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:39.660579 30373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:39.660636 30373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:39.660652 30373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:39.660668 30373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:39.660683 30373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:39.665421 30373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:34891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:34891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:39.666479 30373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:39.667855 30373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:39.676132 30378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:39.676209 30381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:39.676194 30373 server_base.cc:1034] running on GCE node
W20260620 07:20:39.676146 30379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:39.676855 30373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:39.677913 30373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:39.679076 30373 hybrid_clock.cc:648] HybridClock initialized: now 1781940039679058 us; error 30 us; skew 500 ppm
I20260620 07:20:39.680884 30373 webserver.cc:460] Webserver started at http://127.11.123.190:32879/ using document root <none> and password file <none>
I20260620 07:20:39.681636 30373 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:39.681726 30373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:39.681970 30373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:39.684751 30373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "a622e86396c544da9886527cff6a00a2"
format_stamp: "Formatted at 2026-06-20 07:20:39 on dist-test-slave-j663"
server_key: "56b3f9a8493b4de21b967087c84fe1e0"
server_key_iv: "f69abca303f222043a2ff18303a57495"
server_key_version: "encryptionkey@0"
I20260620 07:20:39.685482 30373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "a622e86396c544da9886527cff6a00a2"
format_stamp: "Formatted at 2026-06-20 07:20:39 on dist-test-slave-j663"
server_key: "56b3f9a8493b4de21b967087c84fe1e0"
server_key_iv: "f69abca303f222043a2ff18303a57495"
server_key_version: "encryptionkey@0"
I20260620 07:20:39.688970 30373 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:20:39.691265 30387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:39.692456 30373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:20:39.692551 30373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "a622e86396c544da9886527cff6a00a2"
format_stamp: "Formatted at 2026-06-20 07:20:39 on dist-test-slave-j663"
server_key: "56b3f9a8493b4de21b967087c84fe1e0"
server_key_iv: "f69abca303f222043a2ff18303a57495"
server_key_version: "encryptionkey@0"
I20260620 07:20:39.692631 30373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:39.712917 30373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:39.713587 30373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:39.713785 30373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:39.720777 30373 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:34891
I20260620 07:20:39.720779 30439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:34891 every 8 connection(s)
I20260620 07:20:39.721751 30373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:39.724051 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 30373
I20260620 07:20:39.724190 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:39.724467 11758 external_mini_cluster.cc:1428] Setting key 7c99d382631167c831bc5aade265cbca
I20260620 07:20:39.724856 30440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:39.731750 30440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2: Bootstrap starting.
I20260620 07:20:39.734359 30440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:39.734719 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.725861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.11.123.190:34891)
0620 07:20:39.726508 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:39.726526 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:39.726568 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:20:39.727489 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:39.727512 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:39.728015 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:39.728366 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:39.728808 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.730227 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.731082 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.731752 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.731915 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.732723 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:39.732729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:39.732742 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:39.733269 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:39.733334 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:39.733360 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:39.733453 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:39.733552 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 07:20:39.733814 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
I20260620 07:20:39.735169 30440 log.cc:824] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:39.737371 30440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2: No bootstrap required, opened a new log
I20260620 07:20:39.740806 30440 raft_consensus.cc:348] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a622e86396c544da9886527cff6a00a2" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34891 } }
I20260620 07:20:39.741050 30440 raft_consensus.cc:374] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:39.741146 30440 raft_consensus.cc:729] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a622e86396c544da9886527cff6a00a2, State: Initialized, Role: FOLLOWER
I20260620 07:20:39.741506 30440 consensus_queue.cc:260] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a622e86396c544da9886527cff6a00a2" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34891 } }
I20260620 07:20:39.741614 30440 raft_consensus.cc:388] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:39.741667 30440 raft_consensus.cc:482] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:39.741732 30440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:39.742463 30440 raft_consensus.cc:504] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a622e86396c544da9886527cff6a00a2" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34891 } }
I20260620 07:20:39.742738 30440 leader_election.cc:302] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a622e86396c544da9886527cff6a00a2; no voters:
I20260620 07:20:39.742987 30440 leader_election.cc:288] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:39.743111 30445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:39.743350 30445 raft_consensus.cc:686] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [term 1 LEADER]: Becoming Leader. State: Replica: a622e86396c544da9886527cff6a00a2, State: Running, Role: LEADER
I20260620 07:20:39.743657 30445 consensus_queue.cc:237] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a622e86396c544da9886527cff6a00a2" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34891 } }
I20260620 07:20:39.744221 30440 sys_catalog.cc:564] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:39.745298 30446 sys_catalog.cc:455] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a622e86396c544da9886527cff6a00a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a622e86396c544da9886527cff6a00a2" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34891 } } }
I20260620 07:20:39.745431 30446 sys_catalog.cc:458] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:39.745831 30454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:39.745945 30447 sys_catalog.cc:455] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a622e86396c544da9886527cff6a00a2. Latest consensus state: current_term: 1 leader_uuid: "a622e86396c544da9886527cff6a00a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a622e86396c544da9886527cff6a00a2" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34891 } } }
I20260620 07:20:39.746243 30447 sys_catalog.cc:458] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:39.749953 30454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:39.756307 30454 catalog_manager.cc:1347] Generated new cluster ID: 7f0474f24b9e44279f352c9299cc4b9e
I20260620 07:20:39.756372 30454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:39.769610 30454 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:39.770743 30454 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:39.777480 30454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2: Generated new TSK 0
I20260620 07:20:39.778111 30454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:39.785564 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34891
--builtin_ntp_servers=127.11.123.148:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:20:39.896409 30464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:39.896636 30464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:39.896700 30464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:39.900718 30464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:39.900766 30464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:39.900866 30464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:20:39.905545 30464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:39.906520 30464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:39.907848 30464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:39.917063 30472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:39.917021 30470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:39.917227 30464 server_base.cc:1034] running on GCE node
W20260620 07:20:39.917029 30469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:39.917697 30464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:39.918341 30464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:39.919603 30464 hybrid_clock.cc:648] HybridClock initialized: now 1781940039919566 us; error 47 us; skew 500 ppm
I20260620 07:20:39.921682 30464 webserver.cc:460] Webserver started at http://127.11.123.129:45049/ using document root <none> and password file <none>
I20260620 07:20:39.922304 30464 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:39.922405 30464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:39.922663 30464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:39.925037 30464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "304c1604ee8c4164bd06fe23fac3de59"
format_stamp: "Formatted at 2026-06-20 07:20:39 on dist-test-slave-j663"
server_key: "a41abbe436c3101cd171f2ddc805387a"
server_key_iv: "c9349224dfb569e36b26397775d3c6bd"
server_key_version: "encryptionkey@0"
I20260620 07:20:39.925735 30464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "304c1604ee8c4164bd06fe23fac3de59"
format_stamp: "Formatted at 2026-06-20 07:20:39 on dist-test-slave-j663"
server_key: "a41abbe436c3101cd171f2ddc805387a"
server_key_iv: "c9349224dfb569e36b26397775d3c6bd"
server_key_version: "encryptionkey@0"
I20260620 07:20:39.929409 30464 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:20:39.931957 30478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:39.933341 30464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:20:39.933457 30464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "304c1604ee8c4164bd06fe23fac3de59"
format_stamp: "Formatted at 2026-06-20 07:20:39 on dist-test-slave-j663"
server_key: "a41abbe436c3101cd171f2ddc805387a"
server_key_iv: "c9349224dfb569e36b26397775d3c6bd"
server_key_version: "encryptionkey@0"
I20260620 07:20:39.933604 30464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:39.953720 30464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:39.954417 30464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:39.954632 30464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:39.955346 30464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:39.956820 30464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:39.956903 30464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:39.956975 30464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:39.957018 30464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:39.969029 30464 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:34913
I20260620 07:20:39.969148 30591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:34913 every 8 connection(s)
I20260620 07:20:39.970017 30464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:20:39.971743 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 30464
I20260620 07:20:39.971885 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:20:39.972244 11758 external_mini_cluster.cc:1428] Setting key 8e3091ce1ce93a36fb5bd8f7e22f1250
I20260620 07:20:39.974412 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34891
--builtin_ntp_servers=127.11.123.148:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:20:39.980815 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.972132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:48389 (local address 127.11.123.190:34891)
0620 07:20:39.972285 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:39.972290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:39.973500 (+ 1210us) server_negotiation.cc:407] Connection header received
0620 07:20:39.975409 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:39.975412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:39.975478 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:39.975596 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:39.976619 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.977382 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.978532 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.978968 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.979094 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.980220 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:39.980226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:39.980232 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:39.980250 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:39.980278 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:39.980292 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:39.980386 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:39.980526 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 07:20:39.980606 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:20:39.981571 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:39.972623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:34891 (local address 127.11.123.129:48389)
0620 07:20:39.973366 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:39.973405 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:39.975158 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:39.975814 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:39.975824 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:39.975948 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:39.976474 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.976487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.977514 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.977517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.978403 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:39.978412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:39.979135 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:39.979138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:39.979282 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:39.979940 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:20:39.979941 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:39.979982 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:20:39.980044 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:39.980389 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:39.980395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:39.980400 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:20:39.980491 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:20:39.980707 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260620 07:20:39.982832 30592 heartbeater.cc:347] Connected to a master server at 127.11.123.190:34891
I20260620 07:20:39.983099 30592 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:39.983765 30592 heartbeater.cc:511] Master 127.11.123.190:34891 requested a full tablet report, sending...
I20260620 07:20:39.985669 30404 ts_manager.cc:194] Registered new tserver with Master: 304c1604ee8c4164bd06fe23fac3de59 (127.11.123.129:34913)
I20260620 07:20:39.986856 30404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.123.129:48389
W20260620 07:20:40.086817 30595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:40.087069 30595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:40.087139 30595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:40.091171 30595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:40.091235 30595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:40.091315 30595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:20:40.096782 30595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:40.097874 30595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:40.099171 30595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:40.108356 30603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:40.108330 30600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:40.108347 30601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:40.108726 30595 server_base.cc:1034] running on GCE node
I20260620 07:20:40.109138 30595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:40.109742 30595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:40.110924 30595 hybrid_clock.cc:648] HybridClock initialized: now 1781940040110889 us; error 47 us; skew 500 ppm
I20260620 07:20:40.112560 30595 webserver.cc:460] Webserver started at http://127.11.123.130:45699/ using document root <none> and password file <none>
I20260620 07:20:40.113195 30595 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:40.113288 30595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:40.113517 30595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:40.115432 30595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "806f82cb66dc462a8c09285fa594fc6a"
format_stamp: "Formatted at 2026-06-20 07:20:40 on dist-test-slave-j663"
server_key: "84541674b68291000c6773dd8092708b"
server_key_iv: "8130a3bee1d63ff0f35eef46e46fb426"
server_key_version: "encryptionkey@0"
I20260620 07:20:40.115971 30595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "806f82cb66dc462a8c09285fa594fc6a"
format_stamp: "Formatted at 2026-06-20 07:20:40 on dist-test-slave-j663"
server_key: "84541674b68291000c6773dd8092708b"
server_key_iv: "8130a3bee1d63ff0f35eef46e46fb426"
server_key_version: "encryptionkey@0"
I20260620 07:20:40.119415 30595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:20:40.121858 30609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:40.123080 30595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:20:40.123227 30595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "806f82cb66dc462a8c09285fa594fc6a"
format_stamp: "Formatted at 2026-06-20 07:20:40 on dist-test-slave-j663"
server_key: "84541674b68291000c6773dd8092708b"
server_key_iv: "8130a3bee1d63ff0f35eef46e46fb426"
server_key_version: "encryptionkey@0"
I20260620 07:20:40.123339 30595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:40.135149 30595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:40.135744 30595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:40.135938 30595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:40.136539 30595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:40.137537 30595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:40.137612 30595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:40.137681 30595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:40.137722 30595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:40.146365 30595 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:33035
I20260620 07:20:40.146397 30722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:33035 every 8 connection(s)
I20260620 07:20:40.147205 30595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:20:40.150979 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 30595
I20260620 07:20:40.151084 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:20:40.151432 11758 external_mini_cluster.cc:1428] Setting key ae7e3c5e9ca8bb2a264d59f7aab85aa1
I20260620 07:20:40.153258 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34891
--builtin_ntp_servers=127.11.123.148:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:20:40.157121 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.148960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:57251 (local address 127.11.123.190:34891)
0620 07:20:40.149170 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:40.149175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:40.150188 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 07:20:40.151648 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:40.151651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:40.151709 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:40.151797 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:40.152640 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.153253 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.154980 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.155368 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.155482 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.156529 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:40.156536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:40.156543 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:40.156559 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:40.156586 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:40.156599 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:40.156682 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:40.156842 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 07:20:40.156914 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:20:40.158020 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.149407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:34891 (local address 127.11.123.130:57251)
0620 07:20:40.150041 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:40.150080 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:40.151413 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:40.151959 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:40.151966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:40.152044 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:40.152512 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.152525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.153381 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.153385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.154860 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.154870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.155515 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.155517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.155647 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.156276 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:20:40.156277 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:40.156315 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:20:40.156375 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:40.156704 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:40.156710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:40.156714 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:20:40.156806 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 07:20:40.157072 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260620 07:20:40.159144 30723 heartbeater.cc:347] Connected to a master server at 127.11.123.190:34891
I20260620 07:20:40.159406 30723 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:40.160064 30723 heartbeater.cc:511] Master 127.11.123.190:34891 requested a full tablet report, sending...
I20260620 07:20:40.161345 30404 ts_manager.cc:194] Registered new tserver with Master: 806f82cb66dc462a8c09285fa594fc6a (127.11.123.130:33035)
I20260620 07:20:40.162216 30404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.123.130:57251
W20260620 07:20:40.266291 30726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:40.266605 30726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:40.266678 30726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:40.270750 30726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:40.270824 30726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:40.270910 30726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:20:40.275604 30726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:40.276639 30726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:40.278133 30726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:40.287880 30732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:40.287902 30731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:40.287988 30726 server_base.cc:1034] running on GCE node
W20260620 07:20:40.287886 30734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:40.288483 30726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:40.289068 30726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:40.290290 30726 hybrid_clock.cc:648] HybridClock initialized: now 1781940040290278 us; error 25 us; skew 500 ppm
I20260620 07:20:40.291970 30726 webserver.cc:460] Webserver started at http://127.11.123.131:35367/ using document root <none> and password file <none>
I20260620 07:20:40.292531 30726 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:40.292596 30726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:40.292865 30726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:40.295094 30726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "e9164f4b3e4d4e06a3120903c3046104"
format_stamp: "Formatted at 2026-06-20 07:20:40 on dist-test-slave-j663"
server_key: "b3ba5a3b8c611e2a9389a336c34c11ea"
server_key_iv: "724bb550421613bfa3e2f15b7b721b79"
server_key_version: "encryptionkey@0"
I20260620 07:20:40.295750 30726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "e9164f4b3e4d4e06a3120903c3046104"
format_stamp: "Formatted at 2026-06-20 07:20:40 on dist-test-slave-j663"
server_key: "b3ba5a3b8c611e2a9389a336c34c11ea"
server_key_iv: "724bb550421613bfa3e2f15b7b721b79"
server_key_version: "encryptionkey@0"
I20260620 07:20:40.300961 30726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:20:40.304364 30740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:40.305742 30726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:20:40.305835 30726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "e9164f4b3e4d4e06a3120903c3046104"
format_stamp: "Formatted at 2026-06-20 07:20:40 on dist-test-slave-j663"
server_key: "b3ba5a3b8c611e2a9389a336c34c11ea"
server_key_iv: "724bb550421613bfa3e2f15b7b721b79"
server_key_version: "encryptionkey@0"
I20260620 07:20:40.305918 30726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:40.337219 30726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:40.338093 30726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:40.338277 30726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:40.338855 30726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:40.339807 30726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:40.339859 30726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:40.339900 30726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:40.339919 30726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:40.349015 30726 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:43751
I20260620 07:20:40.349042 30853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:43751 every 8 connection(s)
I20260620 07:20:40.349959 30726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:20:40.358532 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.351623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:54887 (local address 127.11.123.190:34891)
0620 07:20:40.351758 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:40.351761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:40.352738 (+ 977us) server_negotiation.cc:407] Connection header received
0620 07:20:40.354139 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:40.354142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:40.354188 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:40.354259 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:40.355062 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.355564 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.356627 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.356842 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.356924 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.358054 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:40.358057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:40.358059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:40.358069 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:40.358086 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:40.358093 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:40.358180 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:40.358376 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 07:20:40.358422 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:20:40.359335 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.351998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:34891 (local address 127.11.123.131:54887)
0620 07:20:40.352616 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:40.352655 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:40.353956 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:40.354412 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:40.354418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:40.354495 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:40.354945 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.354956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.355670 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.355673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.356525 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.356533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.356971 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.356974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.357171 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.357833 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:20:40.357835 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:40.357872 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:20:40.357928 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:40.358228 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:40.358236 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:40.358241 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:20:40.358351 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 07:20:40.358556 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":109,"threads_started":1}
I20260620 07:20:40.359969 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 30726
I20260620 07:20:40.360100 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:20:40.360389 11758 external_mini_cluster.cc:1428] Setting key 99907011a64b3400b9a3891ce9663bc0
I20260620 07:20:40.360488 30854 heartbeater.cc:347] Connected to a master server at 127.11.123.190:34891
I20260620 07:20:40.360739 30854 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:40.361368 30854 heartbeater.cc:511] Master 127.11.123.190:34891 requested a full tablet report, sending...
I20260620 07:20:40.362447 30404 ts_manager.cc:194] Registered new tserver with Master: e9164f4b3e4d4e06a3120903c3046104 (127.11.123.131:43751)
I20260620 07:20:40.362998 30404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.123.131:54887
I20260620 07:20:40.374255 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:20:40.380147 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.376470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.11.123.190:34891)
0620 07:20:40.376634 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:40.376638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:40.376726 (+ 88us) server_negotiation.cc:407] Connection header received
0620 07:20:40.376825 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:40.376827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:40.376867 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:40.376934 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:40.377255 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.377760 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.378535 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.378749 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.378823 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.379648 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:40.379651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:40.379653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:40.379663 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:40.379680 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:40.379697 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:40.379785 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:40.379915 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 07:20:40.379978 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:20:40.382988 30404 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:20:40.385986 30404 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:20:40.400338 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.394932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:33035 (local address 127.0.0.1:53652)
0620 07:20:40.395483 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:40.395497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:40.395577 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:40.396276 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:40.396280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:40.396302 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:40.396400 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.396406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.397662 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.397667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.398708 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.398717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.399780 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.399783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.399900 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.399942 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 07:20:40.400050 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 07:20:40.400173 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":89,"threads_started":1}
I20260620 07:20:40.400377 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.395371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.11.123.130:33035)
0620 07:20:40.395779 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:40.395790 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:40.395811 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:20:40.395880 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:40.395884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:40.396071 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:40.396309 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:40.396489 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.397547 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.398833 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.399640 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.399790 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.399965 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:40.400093 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 07:20:40.400192 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":108,"threads_started":1}
I20260620 07:20:40.400681 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.394411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.11.123.131:43751)
0620 07:20:40.395301 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:40.395314 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:40.395338 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:20:40.395413 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:40.395418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:40.395629 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:40.395792 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:40.396089 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.397276 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.398987 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.399738 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.399836 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.399965 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:40.400422 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 07:20:40.400521 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":146,"threads_started":1}
I20260620 07:20:40.400718 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.394265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:43751 (local address 127.0.0.1:40088)
0620 07:20:40.394932 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:40.394945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:40.395033 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:40.395797 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:40.395802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:40.395825 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:40.395962 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.395970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.397389 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.397392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.398863 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.398873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.400061 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.400065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.400152 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.400175 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:20:40.400257 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:20:40.400345 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":6,"thread_start_us":124,"threads_started":1}
I20260620 07:20:40.401232 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.394261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:34913 (local address 127.0.0.1:40440)
0620 07:20:40.394627 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:40.394671 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:40.394783 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:40.396528 (+ 1745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:40.396533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:40.396549 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:40.396627 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.396631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.398023 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.398026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.399084 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.399090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.400118 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.400122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.400200 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.400222 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:20:40.400288 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 07:20:40.401114 (+ 826us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":711,"thread_start_us":62,"threads_started":1}
I20260620 07:20:40.401648 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.394997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.11.123.129:34913)
0620 07:20:40.395910 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:40.395925 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:40.395950 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:20:40.396026 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:40.396034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:40.396240 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:40.396380 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:40.396741 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.397893 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.399196 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.399787 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.399904 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.400061 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:40.400371 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 07:20:40.401494 (+ 1123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":140,"threads_started":1}
I20260620 07:20:40.403329 30657 tablet_service.cc:1467] Processing CreateTablet for tablet 9af9d367f6224185880e1f5079bbea58 (DEFAULT_TABLE table=test-table [id=017608bc020b44d196388e6e97ba89e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:40.403391 30788 tablet_service.cc:1467] Processing CreateTablet for tablet 9af9d367f6224185880e1f5079bbea58 (DEFAULT_TABLE table=test-table [id=017608bc020b44d196388e6e97ba89e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:40.404296 30657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af9d367f6224185880e1f5079bbea58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:40.404592 30788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af9d367f6224185880e1f5079bbea58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:40.404433 30526 tablet_service.cc:1467] Processing CreateTablet for tablet 9af9d367f6224185880e1f5079bbea58 (DEFAULT_TABLE table=test-table [id=017608bc020b44d196388e6e97ba89e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:40.405378 30526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af9d367f6224185880e1f5079bbea58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:40.410938 30871 tablet_bootstrap.cc:492] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a: Bootstrap starting.
I20260620 07:20:40.411931 30873 tablet_bootstrap.cc:492] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59: Bootstrap starting.
I20260620 07:20:40.412608 30872 tablet_bootstrap.cc:492] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104: Bootstrap starting.
I20260620 07:20:40.413348 30871 tablet_bootstrap.cc:654] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:40.413991 30873 tablet_bootstrap.cc:654] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:40.414206 30871 log.cc:824] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:40.414402 30872 tablet_bootstrap.cc:654] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:40.414779 30873 log.cc:824] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:40.415220 30872 log.cc:824] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:40.416356 30871 tablet_bootstrap.cc:492] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a: No bootstrap required, opened a new log
I20260620 07:20:40.416543 30871 ts_tablet_manager.cc:1397] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:20:40.416904 30873 tablet_bootstrap.cc:492] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59: No bootstrap required, opened a new log
I20260620 07:20:40.417040 30872 tablet_bootstrap.cc:492] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104: No bootstrap required, opened a new log
I20260620 07:20:40.417126 30873 ts_tablet_manager.cc:1397] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:20:40.417270 30872 ts_tablet_manager.cc:1397] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 07:20:40.420166 30872 raft_consensus.cc:348] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9164f4b3e4d4e06a3120903c3046104" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43751 } } peers { permanent_uuid: "304c1604ee8c4164bd06fe23fac3de59" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34913 } } peers { permanent_uuid: "806f82cb66dc462a8c09285fa594fc6a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 33035 } }
I20260620 07:20:40.420392 30872 raft_consensus.cc:374] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:40.420442 30872 raft_consensus.cc:729] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9164f4b3e4d4e06a3120903c3046104, State: Initialized, Role: FOLLOWER
I20260620 07:20:40.420873 30872 consensus_queue.cc:260] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9164f4b3e4d4e06a3120903c3046104" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43751 } } peers { permanent_uuid: "304c1604ee8c4164bd06fe23fac3de59" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34913 } } peers { permanent_uuid: "806f82cb66dc462a8c09285fa594fc6a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 33035 } }
I20260620 07:20:40.420789 30871 raft_consensus.cc:348] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9164f4b3e4d4e06a3120903c3046104" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43751 } } peers { permanent_uuid: "304c1604ee8c4164bd06fe23fac3de59" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34913 } } peers { permanent_uuid: "806f82cb66dc462a8c09285fa594fc6a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 33035 } }
I20260620 07:20:40.421015 30871 raft_consensus.cc:374] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:40.421072 30871 raft_consensus.cc:729] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806f82cb66dc462a8c09285fa594fc6a, State: Initialized, Role: FOLLOWER
I20260620 07:20:40.421108 30873 raft_consensus.cc:348] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9164f4b3e4d4e06a3120903c3046104" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43751 } } peers { permanent_uuid: "304c1604ee8c4164bd06fe23fac3de59" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34913 } } peers { permanent_uuid: "806f82cb66dc462a8c09285fa594fc6a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 33035 } }
I20260620 07:20:40.421329 30873 raft_consensus.cc:374] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:40.421387 30873 raft_consensus.cc:729] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304c1604ee8c4164bd06fe23fac3de59, State: Initialized, Role: FOLLOWER
I20260620 07:20:40.421501 30871 consensus_queue.cc:260] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9164f4b3e4d4e06a3120903c3046104" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43751 } } peers { permanent_uuid: "304c1604ee8c4164bd06fe23fac3de59" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34913 } } peers { permanent_uuid: "806f82cb66dc462a8c09285fa594fc6a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 33035 } }
I20260620 07:20:40.421765 30854 heartbeater.cc:503] Master 127.11.123.190:34891 was elected leader, sending a full tablet report...
I20260620 07:20:40.422008 30872 ts_tablet_manager.cc:1428] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260620 07:20:40.422400 30871 ts_tablet_manager.cc:1428] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 07:20:40.422747 30723 heartbeater.cc:503] Master 127.11.123.190:34891 was elected leader, sending a full tablet report...
I20260620 07:20:40.423066 30873 consensus_queue.cc:260] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9164f4b3e4d4e06a3120903c3046104" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43751 } } peers { permanent_uuid: "304c1604ee8c4164bd06fe23fac3de59" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34913 } } peers { permanent_uuid: "806f82cb66dc462a8c09285fa594fc6a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 33035 } }
I20260620 07:20:40.423791 30592 heartbeater.cc:503] Master 127.11.123.190:34891 was elected leader, sending a full tablet report...
I20260620 07:20:40.424028 30873 ts_tablet_manager.cc:1428] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 07:20:40.460121 30877 raft_consensus.cc:482] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:20:40.460299 30877 raft_consensus.cc:504] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9164f4b3e4d4e06a3120903c3046104" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43751 } } peers { permanent_uuid: "304c1604ee8c4164bd06fe23fac3de59" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34913 } } peers { permanent_uuid: "806f82cb66dc462a8c09285fa594fc6a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 33035 } }
I20260620 07:20:40.461386 30877 leader_election.cc:288] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 304c1604ee8c4164bd06fe23fac3de59 (127.11.123.129:34913), 806f82cb66dc462a8c09285fa594fc6a (127.11.123.130:33035)
I20260620 07:20:40.464887 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.461554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:34913 (local address 127.11.123.131:58735)
0620 07:20:40.461690 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:40.461708 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:40.461796 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:40.462130 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:40.462133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:40.462151 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:40.462248 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.462253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.463107 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.463111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.463864 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.463872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.464483 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.464487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.464590 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.464608 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 07:20:40.464693 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:20:40.464763 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 07:20:40.464890 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.461762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:58735 (local address 127.11.123.129:34913)
0620 07:20:40.461891 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:40.461896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:40.461912 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:20:40.461964 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:40.461967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:40.462015 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:40.462099 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:40.462357 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.462995 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.463987 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.464365 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.464492 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.464540 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:40.464724 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 07:20:40.464789 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:20:40.465274 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.461754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:55537 (local address 127.11.123.130:33035)
0620 07:20:40.461875 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:40.461880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:40.462346 (+ 466us) server_negotiation.cc:407] Connection header received
0620 07:20:40.462392 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:40.462394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:40.462434 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:40.462488 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:40.462773 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.463230 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.464244 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.464634 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.464768 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.464812 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:40.465051 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 07:20:40.465159 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:20:40.465546 30546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af9d367f6224185880e1f5079bbea58" candidate_uuid: "e9164f4b3e4d4e06a3120903c3046104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304c1604ee8c4164bd06fe23fac3de59" is_pre_election: true
I20260620 07:20:40.465624 30880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.461630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:33035 (local address 127.11.123.131:55537)
0620 07:20:40.462016 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:40.462029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:40.462113 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:40.462533 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:40.462536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:40.462554 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:40.462658 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.462664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.463349 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.463355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.464114 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:40.464122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.464776 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.464780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:40.464884 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.464901 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 07:20:40.465441 (+ 540us) client_negotiation.cc:240] Negotiation successful
0620 07:20:40.465517 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":144,"threads_started":1}
I20260620 07:20:40.465835 30546 raft_consensus.cc:2454] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9164f4b3e4d4e06a3120903c3046104 in term 0.
I20260620 07:20:40.466171 30677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af9d367f6224185880e1f5079bbea58" candidate_uuid: "e9164f4b3e4d4e06a3120903c3046104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806f82cb66dc462a8c09285fa594fc6a" is_pre_election: true
I20260620 07:20:40.466361 30742 leader_election.cc:302] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304c1604ee8c4164bd06fe23fac3de59, e9164f4b3e4d4e06a3120903c3046104; no voters:
I20260620 07:20:40.466421 30677 raft_consensus.cc:2454] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9164f4b3e4d4e06a3120903c3046104 in term 0.
I20260620 07:20:40.466650 30877 raft_consensus.cc:2781] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:20:40.466764 30877 raft_consensus.cc:482] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:20:40.466822 30877 raft_consensus.cc:3037] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:40.467816 30877 raft_consensus.cc:504] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9164f4b3e4d4e06a3120903c3046104" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43751 } } peers { permanent_uuid: "304c1604ee8c4164bd06fe23fac3de59" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34913 } } peers { permanent_uuid: "806f82cb66dc462a8c09285fa594fc6a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 33035 } }
I20260620 07:20:40.468178 30877 leader_election.cc:288] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [CANDIDATE]: Term 1 election: Requested vote from peers 304c1604ee8c4164bd06fe23fac3de59 (127.11.123.129:34913), 806f82cb66dc462a8c09285fa594fc6a (127.11.123.130:33035)
I20260620 07:20:40.468461 30546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af9d367f6224185880e1f5079bbea58" candidate_uuid: "e9164f4b3e4d4e06a3120903c3046104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304c1604ee8c4164bd06fe23fac3de59"
I20260620 07:20:40.468528 30677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af9d367f6224185880e1f5079bbea58" candidate_uuid: "e9164f4b3e4d4e06a3120903c3046104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806f82cb66dc462a8c09285fa594fc6a"
I20260620 07:20:40.468631 30546 raft_consensus.cc:3037] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:40.468647 30677 raft_consensus.cc:3037] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:40.469892 30546 raft_consensus.cc:2454] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9164f4b3e4d4e06a3120903c3046104 in term 1.
I20260620 07:20:40.469916 30677 raft_consensus.cc:2454] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9164f4b3e4d4e06a3120903c3046104 in term 1.
I20260620 07:20:40.470201 30743 leader_election.cc:302] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 806f82cb66dc462a8c09285fa594fc6a, e9164f4b3e4d4e06a3120903c3046104; no voters:
I20260620 07:20:40.470355 30877 raft_consensus.cc:2781] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:40.470554 30877 raft_consensus.cc:686] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 1 LEADER]: Becoming Leader. State: Replica: e9164f4b3e4d4e06a3120903c3046104, State: Running, Role: LEADER
I20260620 07:20:40.470861 30877 consensus_queue.cc:237] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9164f4b3e4d4e06a3120903c3046104" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43751 } } peers { permanent_uuid: "304c1604ee8c4164bd06fe23fac3de59" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34913 } } peers { permanent_uuid: "806f82cb66dc462a8c09285fa594fc6a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 33035 } }
W20260620 07:20:40.471128 30593 tablet.cc:2369] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:20:40.473784 30402 catalog_manager.cc:5495] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9164f4b3e4d4e06a3120903c3046104 (127.11.123.131). New cstate: current_term: 1 leader_uuid: "e9164f4b3e4d4e06a3120903c3046104" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9164f4b3e4d4e06a3120903c3046104" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 43751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "304c1604ee8c4164bd06fe23fac3de59" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "806f82cb66dc462a8c09285fa594fc6a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 33035 } health_report { overall_health: UNKNOWN } } }
I20260620 07:20:40.497187 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.493769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.11.123.131:43751)
0620 07:20:40.493894 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:40.493898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:40.493910 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:20:40.493973 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:40.493975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:40.494022 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:40.494113 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:20:40.494370 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.494844 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.495745 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:40.495951 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:40.496031 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:40.496289 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:20:40.496750 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:20:40.496855 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:40.496979 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 07:20:40.497007 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:20:40.502892 30546 raft_consensus.cc:1261] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59 [term 1 FOLLOWER]: Refusing update from remote peer e9164f4b3e4d4e06a3120903c3046104: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:20:40.502925 30677 raft_consensus.cc:1261] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a [term 1 FOLLOWER]: Refusing update from remote peer e9164f4b3e4d4e06a3120903c3046104: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:20:40.503566 30877 consensus_queue.cc:1035] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [LEADER]: Connected to new peer: Peer: permanent_uuid: "304c1604ee8c4164bd06fe23fac3de59" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:20:40.503755 30881 consensus_queue.cc:1035] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806f82cb66dc462a8c09285fa594fc6a" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 33035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:20:40.514250 30884 mvcc.cc:205] Tried to move back new op lower bound from 7298826405893070848 to 7298826405771460608. Current Snapshot: MvccSnapshot[applied={T|T < 7298826405893070848}]
I20260620 07:20:40.522481 30402 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:20:40.522634 30402 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:20:40.524811 30402 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a622e86396c544da9886527cff6a00a2: Sending DeleteTablet for 3 replicas of tablet 9af9d367f6224185880e1f5079bbea58
I20260620 07:20:40.525625 30526 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af9d367f6224185880e1f5079bbea58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:40 UTC) from {username='slave'} at 127.0.0.1:40440
I20260620 07:20:40.525731 30657 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af9d367f6224185880e1f5079bbea58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:40 UTC) from {username='slave'} at 127.0.0.1:53652
I20260620 07:20:40.525741 30788 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af9d367f6224185880e1f5079bbea58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:40 UTC) from {username='slave'} at 127.0.0.1:40088
I20260620 07:20:40.526139 30894 tablet_replica.cc:330] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a: stopping tablet replica
I20260620 07:20:40.526194 30895 tablet_replica.cc:330] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104: stopping tablet replica
I20260620 07:20:40.526429 30894 raft_consensus.cc:2229] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:20:40.526474 30895 raft_consensus.cc:2229] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:20:40.526719 30894 raft_consensus.cc:2258] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:20:40.526870 30895 raft_consensus.cc:2258] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:20:40.527057 30893 tablet_replica.cc:330] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59: stopping tablet replica
I20260620 07:20:40.527204 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 30464
I20260620 07:20:40.527325 30893 raft_consensus.cc:2229] T 9af9d367f6224185880e1f5079bbea58 P 304c1604ee8c4164bd06fe23fac3de59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:20:40.528445 30895 ts_tablet_manager.cc:1905] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:20:40.530889 30895 ts_tablet_manager.cc:1918] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:20:40.530987 30895 log.cc:1196] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/9af9d367f6224185880e1f5079bbea58
I20260620 07:20:40.531345 30895 ts_tablet_manager.cc:1939] T 9af9d367f6224185880e1f5079bbea58 P e9164f4b3e4d4e06a3120903c3046104: Deleting consensus metadata
I20260620 07:20:40.531587 30894 ts_tablet_manager.cc:1905] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:20:40.532476 30390 catalog_manager.cc:4841] TS e9164f4b3e4d4e06a3120903c3046104 (127.11.123.131:43751): tablet 9af9d367f6224185880e1f5079bbea58 (table test-table [id=017608bc020b44d196388e6e97ba89e5]) successfully deleted
I20260620 07:20:40.533911 30894 ts_tablet_manager.cc:1918] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:20:40.533987 30894 log.cc:1196] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/9af9d367f6224185880e1f5079bbea58
I20260620 07:20:40.534261 30894 ts_tablet_manager.cc:1939] T 9af9d367f6224185880e1f5079bbea58 P 806f82cb66dc462a8c09285fa594fc6a: Deleting consensus metadata
I20260620 07:20:40.535207 30390 catalog_manager.cc:4841] TS 806f82cb66dc462a8c09285fa594fc6a (127.11.123.130:33035): tablet 9af9d367f6224185880e1f5079bbea58 (table test-table [id=017608bc020b44d196388e6e97ba89e5]) successfully deleted
W20260620 07:20:40.539376 30389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:34913 (error 108)
I20260620 07:20:40.540230 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 30595
I20260620 07:20:40.540468 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:40.540127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:34913 (local address 127.0.0.1:40448)
0620 07:20:40.540337 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:40.540395 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:34913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260620 07:20:40.541651 30389 catalog_manager.cc:4584] TS 304c1604ee8c4164bd06fe23fac3de59 (127.11.123.129:34913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9af9d367f6224185880e1f5079bbea58: Network error: Client connection negotiation failed: client connection to 127.11.123.129:34913: connect: Connection refused (error 111)
I20260620 07:20:40.545763 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 30726
I20260620 07:20:40.551383 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 30373
2026-06-20T07:20:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1031 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:40 dist-test-slave-j663 krb5kdc[30899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:40 dist-test-slave-j663 krb5kdc[30899](info): set up 2 sockets
Jun 20 07:20:40 dist-test-slave-j663 krb5kdc[30899](info): commencing operation
krb5kdc: starting...
W20260620 07:20:41.589404 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:41 dist-test-slave-j663 krb5kdc[30899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:20:41.706007 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:34631
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:46263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:34631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:41.816494 30912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:41.816794 30912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:41.816864 30912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:41.821003 30912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:41.821108 30912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:41.821157 30912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:41.821190 30912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:41.821245 30912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:41.826574 30912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:46263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:34631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:34631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:41.827621 30912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:41.828614 30912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:41.837447 30917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:41.837432 30918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:41.837452 30920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:41.837596 30912 server_base.cc:1034] running on GCE node
I20260620 07:20:41.838336 30912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:41.839389 30912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:41.840575 30912 hybrid_clock.cc:648] HybridClock initialized: now 1781940041840561 us; error 27 us; skew 500 ppm
Jun 20 07:20:41 dist-test-slave-j663 krb5kdc[30899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940041, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:41.843319 30912 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:20:41.844182 30912 webserver.cc:460] Webserver started at http://127.11.123.190:33161/ using document root <none> and password file <none>
I20260620 07:20:41.844729 30912 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:41.844816 30912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:41.845043 30912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:41.847012 30912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "9d105b53f72744a8b7c107079955b5b8"
format_stamp: "Formatted at 2026-06-20 07:20:41 on dist-test-slave-j663"
server_key: "b20d624a9ec429da1f17d860bf07356d"
server_key_iv: "35414ad56743a2fa742030cd03a01f81"
server_key_version: "encryptionkey@0"
I20260620 07:20:41.847559 30912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "9d105b53f72744a8b7c107079955b5b8"
format_stamp: "Formatted at 2026-06-20 07:20:41 on dist-test-slave-j663"
server_key: "b20d624a9ec429da1f17d860bf07356d"
server_key_iv: "35414ad56743a2fa742030cd03a01f81"
server_key_version: "encryptionkey@0"
I20260620 07:20:41.851190 30912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 07:20:41.853497 30927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:41.854660 30912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:20:41.854803 30912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "9d105b53f72744a8b7c107079955b5b8"
format_stamp: "Formatted at 2026-06-20 07:20:41 on dist-test-slave-j663"
server_key: "b20d624a9ec429da1f17d860bf07356d"
server_key_iv: "35414ad56743a2fa742030cd03a01f81"
server_key_version: "encryptionkey@0"
I20260620 07:20:41.854912 30912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:41.868912 30912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:41.872161 30912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:41.872351 30912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:41.879465 30912 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:34631
I20260620 07:20:41.879489 30979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:34631 every 8 connection(s)
I20260620 07:20:41.880376 30912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:41.881733 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 30912
I20260620 07:20:41.881894 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:41.882181 11758 external_mini_cluster.cc:1428] Setting key 98274860b4ee03f0353df24a952d1f47
I20260620 07:20:41.883570 30980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:41.890377 30980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8: Bootstrap starting.
Jun 20 07:20:41 dist-test-slave-j663 krb5kdc[30899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:20:41.892992 30980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:41.893747 30980 log.cc:824] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:41.895998 30980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8: No bootstrap required, opened a new log
I20260620 07:20:41.897315 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:41.883558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.11.123.190:34631)
0620 07:20:41.884003 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:41.884030 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:41.884073 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:20:41.884888 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:41.884910 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:41.885362 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:41.885730 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:41.886365 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:41.887905 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:41.888710 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:41.889454 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:41.889626 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:41.892376 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:41.892381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:41.892393 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:41.892454 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:41.894471 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:41.894961 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:41.894965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:41.894970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:41.895073 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:41.895373 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:41.895375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:41.895376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:41.895740 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:41.895879 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:41.896249 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 07:20:41.896555 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":125,"threads_started":1}
I20260620 07:20:41.900069 30980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d105b53f72744a8b7c107079955b5b8" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34631 } }
I20260620 07:20:41.900285 30980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:41.900362 30980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d105b53f72744a8b7c107079955b5b8, State: Initialized, Role: FOLLOWER
I20260620 07:20:41.900858 30980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d105b53f72744a8b7c107079955b5b8" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34631 } }
I20260620 07:20:41.901034 30980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:41.901127 30980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:41.901232 30980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:41.902127 30980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d105b53f72744a8b7c107079955b5b8" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34631 } }
I20260620 07:20:41.902459 30980 leader_election.cc:302] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d105b53f72744a8b7c107079955b5b8; no voters:
I20260620 07:20:41.902768 30980 leader_election.cc:288] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:41.902900 30985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:41.903128 30985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [term 1 LEADER]: Becoming Leader. State: Replica: 9d105b53f72744a8b7c107079955b5b8, State: Running, Role: LEADER
I20260620 07:20:41.903475 30985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d105b53f72744a8b7c107079955b5b8" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34631 } }
I20260620 07:20:41.903978 30980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:41.905229 30986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d105b53f72744a8b7c107079955b5b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d105b53f72744a8b7c107079955b5b8" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34631 } } }
I20260620 07:20:41.905238 30987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d105b53f72744a8b7c107079955b5b8. Latest consensus state: current_term: 1 leader_uuid: "9d105b53f72744a8b7c107079955b5b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d105b53f72744a8b7c107079955b5b8" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 34631 } } }
I20260620 07:20:41.905365 30987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:41.905364 30986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:41.905812 30994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:41.908658 30994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:41.914440 30994 catalog_manager.cc:1347] Generated new cluster ID: ad6b2effdfd6430c8116f7686a3a39f4
I20260620 07:20:41.914508 30994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:41.925228 30994 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:41.926436 30994 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:41.932361 30994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8: Generated new TSK 0
I20260620 07:20:41.933229 30994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.123.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
Entry for principal kudu/127.11.123.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab.
I20260620 07:20:41.971989 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34631
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:46263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:42.085638 31006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:42.085912 31006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:42.085988 31006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:42.089903 31006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:42.089984 31006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:42.090070 31006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:20:42.095160 31006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:46263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.129.keytab
--principal=kudu/127.11.123.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:42.096145 31006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:42.097146 31006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:42.106356 31012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:42.106424 31011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:42.106511 31006 server_base.cc:1034] running on GCE node
W20260620 07:20:42.106356 31014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:42.107065 31006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:42.107793 31006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:42.108961 31006 hybrid_clock.cc:648] HybridClock initialized: now 1781940042108933 us; error 42 us; skew 500 ppm
Jun 20 07:20:42 dist-test-slave-j663 krb5kdc[30899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940042, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:42.111903 31006 init.cc:377] Logged in from keytab as kudu/127.11.123.129@KRBTEST.COM (short username kudu)
I20260620 07:20:42.112783 31006 webserver.cc:460] Webserver started at http://127.11.123.129:37859/ using document root <none> and password file <none>
I20260620 07:20:42.113413 31006 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:42.113476 31006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:42.113696 31006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:42.115666 31006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "58a90277ad1e4de093b65730260c3a5e"
format_stamp: "Formatted at 2026-06-20 07:20:42 on dist-test-slave-j663"
server_key: "19b85fbc22f1ea18c8f90f8ed161fedd"
server_key_iv: "d4c9d43b76a85f683f136b582ad38f5e"
server_key_version: "encryptionkey@0"
I20260620 07:20:42.116221 31006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "58a90277ad1e4de093b65730260c3a5e"
format_stamp: "Formatted at 2026-06-20 07:20:42 on dist-test-slave-j663"
server_key: "19b85fbc22f1ea18c8f90f8ed161fedd"
server_key_iv: "d4c9d43b76a85f683f136b582ad38f5e"
server_key_version: "encryptionkey@0"
I20260620 07:20:42.119951 31006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:20:42.122291 31021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:42.123466 31006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:20:42.123595 31006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "58a90277ad1e4de093b65730260c3a5e"
format_stamp: "Formatted at 2026-06-20 07:20:42 on dist-test-slave-j663"
server_key: "19b85fbc22f1ea18c8f90f8ed161fedd"
server_key_iv: "d4c9d43b76a85f683f136b582ad38f5e"
server_key_version: "encryptionkey@0"
I20260620 07:20:42.123735 31006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:42.145848 31006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:42.148996 31006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:42.149214 31006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:42.149757 31006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:42.150534 31006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:42.150578 31006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:42.150619 31006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:42.150633 31006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:42.159571 31006 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:34179
I20260620 07:20:42.159601 31134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:34179 every 8 connection(s)
I20260620 07:20:42.160411 31006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:20:42.168471 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 31006
I20260620 07:20:42.168605 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:20:42.168917 11758 external_mini_cluster.cc:1428] Setting key 3392759608dbc032e2d325a4fb4bd4f7
Jun 20 07:20:42 dist-test-slave-j663 krb5kdc[30899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940042, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.129@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:20:42.175063 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.162170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:37399 (local address 127.11.123.190:34631)
0620 07:20:42.162320 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:42.162326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:42.163236 (+ 910us) server_negotiation.cc:407] Connection header received
0620 07:20:42.164516 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:42.164519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:42.164568 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:42.164640 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:42.166127 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.166955 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.168069 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.168407 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.168516 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.171257 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:42.171260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:42.171262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:42.171284 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:42.172845 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:42.173496 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:42.173499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:42.173500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:42.173575 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:42.173963 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:42.173966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:42.173967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:42.174131 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:42.174207 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:42.174846 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 07:20:42.174946 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:20:42.176290 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.162605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:34631 (local address 127.11.123.129:37399)
0620 07:20:42.163085 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:42.163127 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:42.164326 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:42.164817 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:42.164823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:42.165288 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:42.165970 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.165987 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.167076 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.167079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.167967 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.167976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.168616 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.168623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.168818 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.169471 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:20:42.169495 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:42.171085 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:42.172975 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:42.172983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:42.173000 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:42.173399 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:42.173668 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:42.173675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:42.173680 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:42.173865 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:42.174242 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:42.174251 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:42.174579 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 07:20:42.174793 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 07:20:42.175161 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":122,"threads_started":1}
I20260620 07:20:42.177567 31135 heartbeater.cc:347] Connected to a master server at 127.11.123.190:34631
I20260620 07:20:42.177821 31135 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:42.178431 31135 heartbeater.cc:511] Master 127.11.123.190:34631 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.123.130@KRBTEST.COM; defaulting to no policy
I20260620 07:20:42.180047 30944 ts_manager.cc:194] Registered new tserver with Master: 58a90277ad1e4de093b65730260c3a5e (127.11.123.129:34179)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.130@KRBTEST.COM" created.
I20260620 07:20:42.181370 30944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.129@KRBTEST.COM'} at 127.11.123.129:37399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
Entry for principal kudu/127.11.123.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab.
I20260620 07:20:42.196472 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34631
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:46263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:42.307515 31140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:42.307758 31140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:42.307828 31140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:42.311772 31140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:42.311826 31140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:42.311926 31140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:20:42.317199 31140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:46263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.130.keytab
--principal=kudu/127.11.123.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:42.318198 31140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:42.319204 31140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:42.328791 31145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:42.328747 31148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:42.328876 31140 server_base.cc:1034] running on GCE node
W20260620 07:20:42.328743 31146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:42.329401 31140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:42.330073 31140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:42.331274 31140 hybrid_clock.cc:648] HybridClock initialized: now 1781940042331260 us; error 29 us; skew 500 ppm
Jun 20 07:20:42 dist-test-slave-j663 krb5kdc[30899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940042, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:42.334326 31140 init.cc:377] Logged in from keytab as kudu/127.11.123.130@KRBTEST.COM (short username kudu)
I20260620 07:20:42.335520 31140 webserver.cc:460] Webserver started at http://127.11.123.130:39803/ using document root <none> and password file <none>
I20260620 07:20:42.336314 31140 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:42.336414 31140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:42.336692 31140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:42.339113 31140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "4bc023d63974466face0f4ecd81be427"
format_stamp: "Formatted at 2026-06-20 07:20:42 on dist-test-slave-j663"
server_key: "11f404a4649cd7b76143a22dcbf3d097"
server_key_iv: "f6a69881298b91db49aa9996f94f8b09"
server_key_version: "encryptionkey@0"
I20260620 07:20:42.339635 31140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "4bc023d63974466face0f4ecd81be427"
format_stamp: "Formatted at 2026-06-20 07:20:42 on dist-test-slave-j663"
server_key: "11f404a4649cd7b76143a22dcbf3d097"
server_key_iv: "f6a69881298b91db49aa9996f94f8b09"
server_key_version: "encryptionkey@0"
I20260620 07:20:42.343395 31140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 07:20:42.345811 31155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:42.347070 31140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:20:42.347316 31140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "4bc023d63974466face0f4ecd81be427"
format_stamp: "Formatted at 2026-06-20 07:20:42 on dist-test-slave-j663"
server_key: "11f404a4649cd7b76143a22dcbf3d097"
server_key_iv: "f6a69881298b91db49aa9996f94f8b09"
server_key_version: "encryptionkey@0"
I20260620 07:20:42.347417 31140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:42.362094 31140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:42.365185 31140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:42.365375 31140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:42.365940 31140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:42.366730 31140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:42.366776 31140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:42.366842 31140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:42.366882 31140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:42.376171 31140 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:44383
I20260620 07:20:42.376199 31268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:44383 every 8 connection(s)
I20260620 07:20:42.377030 31140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:20:42.382591 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 31140
I20260620 07:20:42.382723 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:20:42.383001 11758 external_mini_cluster.cc:1428] Setting key 3bde2e8e4eb6fd9d4b698807e1d9fabd
Jun 20 07:20:42 dist-test-slave-j663 krb5kdc[30899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940042, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.130@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:20:42.391204 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.378824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:49257 (local address 127.11.123.190:34631)
0620 07:20:42.378966 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:42.378972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:42.379878 (+ 906us) server_negotiation.cc:407] Connection header received
0620 07:20:42.381151 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:42.381154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:42.381201 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:42.381275 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:42.382420 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.383068 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.384289 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.384511 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.384619 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.387166 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:42.387172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:42.387178 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:42.387210 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:42.388940 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:42.389530 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:42.389536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:42.389541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:42.389642 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:42.389986 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:42.389991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:42.389994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:42.390271 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:42.390394 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:42.390864 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 07:20:42.391010 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:20:42.391944 31271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.379257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:34631 (local address 127.11.123.130:49257)
0620 07:20:42.379733 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:42.379774 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:42.380898 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:42.381429 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:42.381435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:42.381805 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:42.382251 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.382264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.383211 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.383214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.384145 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.384156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.384664 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.384668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.384833 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.385436 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:20:42.385452 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:42.387014 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:42.389063 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:42.389067 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:42.389146 (+ 79us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:42.389414 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:42.389750 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:42.389753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:42.389754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:42.389886 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:42.390391 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:42.390396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:42.390626 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 07:20:42.390778 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 07:20:42.391016 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.11.123.131@KRBTEST.COM; defaulting to no policy
I20260620 07:20:42.393224 31269 heartbeater.cc:347] Connected to a master server at 127.11.123.190:34631
I20260620 07:20:42.393467 31269 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:42.393942 31269 heartbeater.cc:511] Master 127.11.123.190:34631 requested a full tablet report, sending...
I20260620 07:20:42.395107 30944 ts_manager.cc:194] Registered new tserver with Master: 4bc023d63974466face0f4ecd81be427 (127.11.123.130:44383)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.131@KRBTEST.COM" created.
I20260620 07:20:42.395798 30944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.130@KRBTEST.COM'} at 127.11.123.130:49257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
Entry for principal kudu/127.11.123.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab.
I20260620 07:20:42.413456 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34631
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:46263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:42.522696 31274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:42.522974 31274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:42.523046 31274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:42.527132 31274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:42.527226 31274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:42.527351 31274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:20:42.532599 31274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:46263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.131.keytab
--principal=kudu/127.11.123.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:34631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:42.533648 31274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:42.534616 31274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:42.543892 31279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:42.543903 31280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:42.543936 31282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:42.544247 31274 server_base.cc:1034] running on GCE node
I20260620 07:20:42.544634 31274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:42.545286 31274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:42.546463 31274 hybrid_clock.cc:648] HybridClock initialized: now 1781940042546444 us; error 27 us; skew 500 ppm
Jun 20 07:20:42 dist-test-slave-j663 krb5kdc[30899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940042, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:42.549649 31274 init.cc:377] Logged in from keytab as kudu/127.11.123.131@KRBTEST.COM (short username kudu)
I20260620 07:20:42.550468 31274 webserver.cc:460] Webserver started at http://127.11.123.131:39077/ using document root <none> and password file <none>
I20260620 07:20:42.551024 31274 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:42.551082 31274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:42.551338 31274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:42.553342 31274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "b978fea78db549bb8c1f443d4dceaec4"
format_stamp: "Formatted at 2026-06-20 07:20:42 on dist-test-slave-j663"
server_key: "1a95cc8ff0a71b0929a781026d1d4e6e"
server_key_iv: "0068f4e6551bcd71300f03dd04b3687f"
server_key_version: "encryptionkey@0"
I20260620 07:20:42.553874 31274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "b978fea78db549bb8c1f443d4dceaec4"
format_stamp: "Formatted at 2026-06-20 07:20:42 on dist-test-slave-j663"
server_key: "1a95cc8ff0a71b0929a781026d1d4e6e"
server_key_iv: "0068f4e6551bcd71300f03dd04b3687f"
server_key_version: "encryptionkey@0"
I20260620 07:20:42.557374 31274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:20:42.559831 31289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:42.560982 31274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:20:42.561167 31274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "b978fea78db549bb8c1f443d4dceaec4"
format_stamp: "Formatted at 2026-06-20 07:20:42 on dist-test-slave-j663"
server_key: "1a95cc8ff0a71b0929a781026d1d4e6e"
server_key_iv: "0068f4e6551bcd71300f03dd04b3687f"
server_key_version: "encryptionkey@0"
I20260620 07:20:42.561285 31274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:42.573544 31274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:42.576653 31274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:42.576859 31274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:42.577473 31274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:42.578254 31274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:42.578325 31274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:42.578388 31274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:42.578430 31274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:42.587543 31274 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:38967
I20260620 07:20:42.587602 31402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:38967 every 8 connection(s)
I20260620 07:20:42.588447 31274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:20:42.589421 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 31274
I20260620 07:20:42.589557 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:20:42.589871 11758 external_mini_cluster.cc:1428] Setting key 30bfe6a5da8d3123038dab2847376444
Jun 20 07:20:42 dist-test-slave-j663 krb5kdc[30899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940042, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.131@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:20:42.603130 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.590436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:42819 (local address 127.11.123.190:34631)
0620 07:20:42.590583 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:42.590588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:42.592027 (+ 1439us) server_negotiation.cc:407] Connection header received
0620 07:20:42.593402 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:42.593405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:42.593457 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:42.593533 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:42.594641 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.595180 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.596303 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.596536 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.596642 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.599482 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:42.599486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:42.599488 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:42.599517 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:42.600785 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:42.601408 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:42.601412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:42.601414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:42.601501 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:42.601866 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:42.601869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:42.601871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:42.602065 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:42.602194 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:42.602801 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 07:20:42.602956 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:20:42.604120 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.590824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:34631 (local address 127.11.123.131:42819)
0620 07:20:42.591887 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:42.591942 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:42.593209 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:42.593705 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:42.593712 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:42.594037 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:42.594504 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.594518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.595302 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.595312 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.596175 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.596184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.596686 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.596690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.596853 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.597465 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:20:42.597481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:42.599313 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:42.600906 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:42.600912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:42.600926 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:42.601304 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:42.601601 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:20:42.601604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:20:42.601606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:20:42.601760 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:20:42.602193 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:42.602201 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:42.602449 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 07:20:42.602676 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 07:20:42.603040 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":143,"threads_started":1}
I20260620 07:20:42.605330 31403 heartbeater.cc:347] Connected to a master server at 127.11.123.190:34631
I20260620 07:20:42.605609 31403 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:42.606163 31403 heartbeater.cc:511] Master 127.11.123.190:34631 requested a full tablet report, sending...
I20260620 07:20:42.607136 30944 ts_manager.cc:194] Registered new tserver with Master: b978fea78db549bb8c1f443d4dceaec4 (127.11.123.131:38967)
I20260620 07:20:42.607712 30944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.123.131@KRBTEST.COM'} at 127.11.123.131:42819
I20260620 07:20:42.615659 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:20:42.625300 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.617673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45232 (local address 127.11.123.190:34631)
0620 07:20:42.617802 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:42.617806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:42.617950 (+ 144us) server_negotiation.cc:407] Connection header received
0620 07:20:42.618061 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:42.618063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:42.618100 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:42.618167 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:42.619001 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.619538 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.620524 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.620740 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.620821 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.622315 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:42.622319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:42.622321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:42.622350 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:42.623835 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:42.624220 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:42.624222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:42.624223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:42.624293 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:42.624515 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:42.624518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:42.624519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:42.624723 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:42.624822 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:42.625042 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 07:20:42.625184 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:20:42.628333 30944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:20:42.631373 30944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:20:42.645196 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.639239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:44383 (local address 127.0.0.1:33034)
0620 07:20:42.639611 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:42.639662 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:42.639788 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:42.640842 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:42.640847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:42.640871 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:42.641017 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.641024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.642241 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.642247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.643519 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.643528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.644596 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.644600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.644721 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.644765 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 07:20:42.644890 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 07:20:42.645016 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260620 07:20:42.645215 31415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.639367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.11.123.130:44383)
0620 07:20:42.640374 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:42.640388 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:42.640410 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:20:42.640490 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:42.640495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:42.640691 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:42.640848 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:42.641184 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.642111 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.643645 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.644474 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.644608 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.644730 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:42.644921 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 07:20:42.645016 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":111,"threads_started":1}
I20260620 07:20:42.645255 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.639504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:34179 (local address 127.0.0.1:48770)
0620 07:20:42.640024 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:42.640040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:42.640131 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:42.641018 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:42.641021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:42.641037 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:42.641185 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.641193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.642617 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.642620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.643701 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.643707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.644783 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.644786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.644875 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.644897 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:20:42.644971 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 07:20:42.645042 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260620 07:20:42.645669 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.639912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.11.123.129:34179)
0620 07:20:42.640429 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:42.640443 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:42.640460 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:20:42.640524 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:42.640528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:42.640691 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:42.640848 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:42.641316 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.642470 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.643808 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.644478 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.644598 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.644720 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:42.645438 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 07:20:42.645531 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":123,"threads_started":1}
I20260620 07:20:42.646026 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.639978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:38967 (local address 127.0.0.1:54300)
0620 07:20:42.640515 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:42.640529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:42.640617 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:42.641444 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:42.641447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:42.641465 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:42.641557 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.641563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.642997 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.643003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.644196 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.644207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.645647 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.645650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.645764 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.645791 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:20:42.645847 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 07:20:42.645916 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":92,"threads_started":1}
I20260620 07:20:42.646819 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.640162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.11.123.131:38967)
0620 07:20:42.640923 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:42.640936 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:42.640957 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:20:42.641031 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:42.641035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:42.641298 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:42.641452 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:42.641650 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.642714 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.644328 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.645180 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.646445 (+ 1265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.646556 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:42.646614 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 07:20:42.646693 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":112,"threads_started":1}
I20260620 07:20:42.648265 31069 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba9b53cdfa24d1b8f1bb87531823d91 (DEFAULT_TABLE table=test-table [id=e8852caebeca456c89898655b3891aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:42.648305 31203 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba9b53cdfa24d1b8f1bb87531823d91 (DEFAULT_TABLE table=test-table [id=e8852caebeca456c89898655b3891aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:42.649266 31069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba9b53cdfa24d1b8f1bb87531823d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:42.649318 31203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba9b53cdfa24d1b8f1bb87531823d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:42.649160 31337 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba9b53cdfa24d1b8f1bb87531823d91 (DEFAULT_TABLE table=test-table [id=e8852caebeca456c89898655b3891aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:42.650014 31337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba9b53cdfa24d1b8f1bb87531823d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:42.655305 31420 tablet_bootstrap.cc:492] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e: Bootstrap starting.
I20260620 07:20:42.656770 31421 tablet_bootstrap.cc:492] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427: Bootstrap starting.
I20260620 07:20:42.657379 31422 tablet_bootstrap.cc:492] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4: Bootstrap starting.
I20260620 07:20:42.658141 31420 tablet_bootstrap.cc:654] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:42.658792 31421 tablet_bootstrap.cc:654] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:42.659438 31420 log.cc:824] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:42.659785 31422 tablet_bootstrap.cc:654] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:42.659926 31421 log.cc:824] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:42.660979 31422 log.cc:824] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:42.661636 31421 tablet_bootstrap.cc:492] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427: No bootstrap required, opened a new log
I20260620 07:20:42.661650 31420 tablet_bootstrap.cc:492] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e: No bootstrap required, opened a new log
I20260620 07:20:42.661829 31420 ts_tablet_manager.cc:1397] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 07:20:42.661844 31421 ts_tablet_manager.cc:1397] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:20:42.664642 31421 raft_consensus.cc:348] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc023d63974466face0f4ecd81be427" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 44383 } } peers { permanent_uuid: "b978fea78db549bb8c1f443d4dceaec4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 38967 } } peers { permanent_uuid: "58a90277ad1e4de093b65730260c3a5e" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34179 } }
I20260620 07:20:42.664672 31420 raft_consensus.cc:348] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc023d63974466face0f4ecd81be427" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 44383 } } peers { permanent_uuid: "b978fea78db549bb8c1f443d4dceaec4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 38967 } } peers { permanent_uuid: "58a90277ad1e4de093b65730260c3a5e" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34179 } }
I20260620 07:20:42.664919 31421 raft_consensus.cc:374] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:42.664878 31420 raft_consensus.cc:374] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:42.664986 31420 raft_consensus.cc:729] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a90277ad1e4de093b65730260c3a5e, State: Initialized, Role: FOLLOWER
I20260620 07:20:42.665011 31421 raft_consensus.cc:729] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc023d63974466face0f4ecd81be427, State: Initialized, Role: FOLLOWER
I20260620 07:20:42.665491 31420 consensus_queue.cc:260] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc023d63974466face0f4ecd81be427" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 44383 } } peers { permanent_uuid: "b978fea78db549bb8c1f443d4dceaec4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 38967 } } peers { permanent_uuid: "58a90277ad1e4de093b65730260c3a5e" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34179 } }
I20260620 07:20:42.665489 31421 consensus_queue.cc:260] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc023d63974466face0f4ecd81be427" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 44383 } } peers { permanent_uuid: "b978fea78db549bb8c1f443d4dceaec4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 38967 } } peers { permanent_uuid: "58a90277ad1e4de093b65730260c3a5e" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34179 } }
I20260620 07:20:42.666335 31135 heartbeater.cc:503] Master 127.11.123.190:34631 was elected leader, sending a full tablet report...
I20260620 07:20:42.666333 31269 heartbeater.cc:503] Master 127.11.123.190:34631 was elected leader, sending a full tablet report...
I20260620 07:20:42.666496 31421 ts_tablet_manager.cc:1428] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:20:42.666594 31422 tablet_bootstrap.cc:492] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4: No bootstrap required, opened a new log
I20260620 07:20:42.666749 31422 ts_tablet_manager.cc:1397] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20260620 07:20:42.667474 31420 ts_tablet_manager.cc:1428] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:20:42.670428 31422 raft_consensus.cc:348] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc023d63974466face0f4ecd81be427" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 44383 } } peers { permanent_uuid: "b978fea78db549bb8c1f443d4dceaec4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 38967 } } peers { permanent_uuid: "58a90277ad1e4de093b65730260c3a5e" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34179 } }
I20260620 07:20:42.670676 31422 raft_consensus.cc:374] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:42.670737 31422 raft_consensus.cc:729] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b978fea78db549bb8c1f443d4dceaec4, State: Initialized, Role: FOLLOWER
I20260620 07:20:42.671140 31422 consensus_queue.cc:260] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc023d63974466face0f4ecd81be427" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 44383 } } peers { permanent_uuid: "b978fea78db549bb8c1f443d4dceaec4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 38967 } } peers { permanent_uuid: "58a90277ad1e4de093b65730260c3a5e" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34179 } }
I20260620 07:20:42.671823 31403 heartbeater.cc:503] Master 127.11.123.190:34631 was elected leader, sending a full tablet report...
I20260620 07:20:42.672046 31422 ts_tablet_manager.cc:1428] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 07:20:42.722797 31427 raft_consensus.cc:482] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:20:42.723042 31427 raft_consensus.cc:504] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc023d63974466face0f4ecd81be427" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 44383 } } peers { permanent_uuid: "b978fea78db549bb8c1f443d4dceaec4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 38967 } } peers { permanent_uuid: "58a90277ad1e4de093b65730260c3a5e" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34179 } }
I20260620 07:20:42.724243 31427 leader_election.cc:288] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b978fea78db549bb8c1f443d4dceaec4 (127.11.123.131:38967), 58a90277ad1e4de093b65730260c3a5e (127.11.123.129:34179)
I20260620 07:20:42.729204 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.724726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:49109 (local address 127.11.123.129:34179)
0620 07:20:42.724869 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:42.724873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:42.724991 (+ 118us) server_negotiation.cc:407] Connection header received
0620 07:20:42.725064 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:42.725067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:42.725155 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:42.725243 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:42.725571 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.726280 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.727848 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.728475 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.728602 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.728654 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:42.728986 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 07:20:42.729051 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:20:42.729204 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.724556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:42707 (local address 127.11.123.131:38967)
0620 07:20:42.724710 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:42.724714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:42.724731 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:20:42.724855 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:42.724858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:42.724917 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:42.724993 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:42.725377 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.726172 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.727848 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.728496 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.728646 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.728706 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:42.728986 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 07:20:42.729054 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:20:42.729272 31271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.724414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:38967 (local address 127.11.123.130:42707)
0620 07:20:42.724609 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:42.724628 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:42.724722 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:42.725030 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:42.725034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:42.725056 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:42.725212 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.725244 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.726295 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.726297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.727720 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.727730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.728643 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.728646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.728796 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.728869 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 07:20:42.728954 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:20:42.729121 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":82}
I20260620 07:20:42.729281 31429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.724526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:34179 (local address 127.11.123.130:49109)
0620 07:20:42.724877 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:42.724888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:42.724987 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:42.725286 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:42.725292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:42.725313 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:42.725443 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.725451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.726379 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.726382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.727720 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:42.727730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.728630 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.728634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:42.728799 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.728874 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 07:20:42.728954 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:20:42.729121 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":175,"thread_start_us":167,"threads_started":1}
I20260620 07:20:42.729943 31357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba9b53cdfa24d1b8f1bb87531823d91" candidate_uuid: "4bc023d63974466face0f4ecd81be427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b978fea78db549bb8c1f443d4dceaec4" is_pre_election: true
I20260620 07:20:42.730121 31089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba9b53cdfa24d1b8f1bb87531823d91" candidate_uuid: "4bc023d63974466face0f4ecd81be427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a90277ad1e4de093b65730260c3a5e" is_pre_election: true
I20260620 07:20:42.730281 31357 raft_consensus.cc:2454] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc023d63974466face0f4ecd81be427 in term 0.
I20260620 07:20:42.730369 31089 raft_consensus.cc:2454] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc023d63974466face0f4ecd81be427 in term 0.
I20260620 07:20:42.730818 31157 leader_election.cc:302] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc023d63974466face0f4ecd81be427, 58a90277ad1e4de093b65730260c3a5e; no voters:
I20260620 07:20:42.731102 31427 raft_consensus.cc:2781] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:20:42.731204 31427 raft_consensus.cc:482] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:20:42.731264 31427 raft_consensus.cc:3037] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:42.732331 31427 raft_consensus.cc:504] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc023d63974466face0f4ecd81be427" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 44383 } } peers { permanent_uuid: "b978fea78db549bb8c1f443d4dceaec4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 38967 } } peers { permanent_uuid: "58a90277ad1e4de093b65730260c3a5e" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34179 } }
I20260620 07:20:42.732666 31427 leader_election.cc:288] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [CANDIDATE]: Term 1 election: Requested vote from peers b978fea78db549bb8c1f443d4dceaec4 (127.11.123.131:38967), 58a90277ad1e4de093b65730260c3a5e (127.11.123.129:34179)
I20260620 07:20:42.733067 31089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba9b53cdfa24d1b8f1bb87531823d91" candidate_uuid: "4bc023d63974466face0f4ecd81be427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a90277ad1e4de093b65730260c3a5e"
I20260620 07:20:42.733206 31357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba9b53cdfa24d1b8f1bb87531823d91" candidate_uuid: "4bc023d63974466face0f4ecd81be427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b978fea78db549bb8c1f443d4dceaec4"
I20260620 07:20:42.733235 31089 raft_consensus.cc:3037] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:42.733330 31357 raft_consensus.cc:3037] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:42.734558 31089 raft_consensus.cc:2454] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc023d63974466face0f4ecd81be427 in term 1.
I20260620 07:20:42.734552 31357 raft_consensus.cc:2454] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc023d63974466face0f4ecd81be427 in term 1.
I20260620 07:20:42.734830 31157 leader_election.cc:302] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc023d63974466face0f4ecd81be427, 58a90277ad1e4de093b65730260c3a5e; no voters:
I20260620 07:20:42.735018 31427 raft_consensus.cc:2781] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:42.735292 31427 raft_consensus.cc:686] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 1 LEADER]: Becoming Leader. State: Replica: 4bc023d63974466face0f4ecd81be427, State: Running, Role: LEADER
I20260620 07:20:42.735632 31427 consensus_queue.cc:237] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc023d63974466face0f4ecd81be427" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 44383 } } peers { permanent_uuid: "b978fea78db549bb8c1f443d4dceaec4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 38967 } } peers { permanent_uuid: "58a90277ad1e4de093b65730260c3a5e" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34179 } }
I20260620 07:20:42.738555 30944 catalog_manager.cc:5495] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bc023d63974466face0f4ecd81be427 (127.11.123.130). New cstate: current_term: 1 leader_uuid: "4bc023d63974466face0f4ecd81be427" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc023d63974466face0f4ecd81be427" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 44383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b978fea78db549bb8c1f443d4dceaec4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58a90277ad1e4de093b65730260c3a5e" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34179 } health_report { overall_health: UNKNOWN } } }
I20260620 07:20:42.767963 31415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:42.764269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.11.123.130:44383)
0620 07:20:42.764395 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:42.764400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:42.764416 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:20:42.764542 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:42.764545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:42.764602 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:42.764719 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:20:42.764978 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.765591 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.766473 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:42.766697 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:42.766805 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:42.767135 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:20:42.767548 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:20:42.767683 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:42.767746 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 07:20:42.767802 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 07:20:42.773767 31089 raft_consensus.cc:1261] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e [term 1 FOLLOWER]: Refusing update from remote peer 4bc023d63974466face0f4ecd81be427: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:20:42.773967 31357 raft_consensus.cc:1261] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4 [term 1 FOLLOWER]: Refusing update from remote peer 4bc023d63974466face0f4ecd81be427: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:20:42.774535 31427 consensus_queue.cc:1035] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a90277ad1e4de093b65730260c3a5e" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:20:42.774791 31430 consensus_queue.cc:1035] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b978fea78db549bb8c1f443d4dceaec4" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:20:42.783562 31435 mvcc.cc:205] Tried to move back new op lower bound from 7298826415194001408 to 7298826415047503872. Current Snapshot: MvccSnapshot[applied={T|T < 7298826415194001408}]
I20260620 07:20:42.790099 30944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:20:42.790285 30944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:20:42.792589 30944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d105b53f72744a8b7c107079955b5b8: Sending DeleteTablet for 3 replicas of tablet 8ba9b53cdfa24d1b8f1bb87531823d91
I20260620 07:20:42.793455 31069 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ba9b53cdfa24d1b8f1bb87531823d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:42 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:48770
I20260620 07:20:42.793715 31203 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ba9b53cdfa24d1b8f1bb87531823d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:42 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:33034
I20260620 07:20:42.793949 31443 tablet_replica.cc:330] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e: stopping tablet replica
I20260620 07:20:42.793932 31337 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ba9b53cdfa24d1b8f1bb87531823d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:42 UTC) from {username='kudu', principal='kudu/127.11.123.190@KRBTEST.COM'} at 127.0.0.1:54300
I20260620 07:20:42.794142 31444 tablet_replica.cc:330] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427: stopping tablet replica
I20260620 07:20:42.794253 31443 raft_consensus.cc:2229] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:20:42.794369 31444 raft_consensus.cc:2229] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:20:42.794597 31443 raft_consensus.cc:2258] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 58a90277ad1e4de093b65730260c3a5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:20:42.794749 31444 raft_consensus.cc:2258] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:20:42.794873 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 31006
I20260620 07:20:42.798565 31444 ts_tablet_manager.cc:1905] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:20:42.800761 31444 ts_tablet_manager.cc:1918] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:20:42.800875 31444 log.cc:1196] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/8ba9b53cdfa24d1b8f1bb87531823d91
I20260620 07:20:42.801234 31445 tablet_replica.cc:330] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4: stopping tablet replica
I20260620 07:20:42.801425 31444 ts_tablet_manager.cc:1939] T 8ba9b53cdfa24d1b8f1bb87531823d91 P 4bc023d63974466face0f4ecd81be427: Deleting consensus metadata
I20260620 07:20:42.801584 31445 raft_consensus.cc:2229] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:20:42.801910 31445 raft_consensus.cc:2258] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 07:20:42.802539 30929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:34179 (error 108)
I20260620 07:20:42.803196 30931 catalog_manager.cc:4841] TS 4bc023d63974466face0f4ecd81be427 (127.11.123.130:44383): tablet 8ba9b53cdfa24d1b8f1bb87531823d91 (table test-table [id=e8852caebeca456c89898655b3891aa1]) successfully deleted
I20260620 07:20:42.803381 31445 ts_tablet_manager.cc:1905] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 07:20:42.803902 31418 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:20:42.803581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:34179 (local address 127.0.0.1:48782)
0620 07:20:42.803721 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:42.803803 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:34179: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 07:20:42.804105 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 31140
W20260620 07:20:42.804179 30929 catalog_manager.cc:4584] TS 58a90277ad1e4de093b65730260c3a5e (127.11.123.129:34179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ba9b53cdfa24d1b8f1bb87531823d91: Network error: Client connection negotiation failed: client connection to 127.11.123.129:34179: connect: Connection reset by peer (error 104)
I20260620 07:20:42.806078 31445 ts_tablet_manager.cc:1918] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:20:42.806188 31445 log.cc:1196] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/8ba9b53cdfa24d1b8f1bb87531823d91
I20260620 07:20:42.806484 31445 ts_tablet_manager.cc:1939] T 8ba9b53cdfa24d1b8f1bb87531823d91 P b978fea78db549bb8c1f443d4dceaec4: Deleting consensus metadata
I20260620 07:20:42.807499 30931 catalog_manager.cc:4841] TS b978fea78db549bb8c1f443d4dceaec4 (127.11.123.131:38967): tablet 8ba9b53cdfa24d1b8f1bb87531823d91 (table test-table [id=e8852caebeca456c89898655b3891aa1]) successfully deleted
I20260620 07:20:42.810510 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 31274
I20260620 07:20:42.816262 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 30912
2026-06-20T07:20:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2265 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T07:20:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:42Z Disabled control of system clock
I20260620 07:20:42.846519 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:32871
--webserver_interface=127.11.123.190
--webserver_port=0
--builtin_ntp_servers=127.11.123.148:39005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:32871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:20:42.957171 31450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:42.957460 31450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:42.957537 31450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:42.963265 31450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:42.963351 31450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:42.963378 31450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:42.963415 31450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:42.963447 31450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:42.968359 31450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:32871
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:32871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:42.969444 31450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:42.970767 31450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:42.978891 31456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:42.978911 31458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:42.979009 31450 server_base.cc:1034] running on GCE node
W20260620 07:20:42.978940 31455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:42.979722 31450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:42.980809 31450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:42.982012 31450 hybrid_clock.cc:648] HybridClock initialized: now 1781940042981967 us; error 54 us; skew 500 ppm
I20260620 07:20:42.983958 31450 webserver.cc:460] Webserver started at http://127.11.123.190:34341/ using document root <none> and password file <none>
I20260620 07:20:42.984553 31450 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:42.984653 31450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:42.984902 31450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:42.987068 31450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "0fc365de1c034457a981729bc7ea746d"
format_stamp: "Formatted at 2026-06-20 07:20:42 on dist-test-slave-j663"
server_key: "809a44c7066f79e3a59ea275d4e70258"
server_key_iv: "2426154db2dc6d150abfc88e9a98b499"
server_key_version: "encryptionkey@0"
I20260620 07:20:42.987651 31450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "0fc365de1c034457a981729bc7ea746d"
format_stamp: "Formatted at 2026-06-20 07:20:42 on dist-test-slave-j663"
server_key: "809a44c7066f79e3a59ea275d4e70258"
server_key_iv: "2426154db2dc6d150abfc88e9a98b499"
server_key_version: "encryptionkey@0"
I20260620 07:20:42.991374 31450 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 07:20:42.994067 31464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:42.995304 31450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:20:42.995460 31450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "0fc365de1c034457a981729bc7ea746d"
format_stamp: "Formatted at 2026-06-20 07:20:42 on dist-test-slave-j663"
server_key: "809a44c7066f79e3a59ea275d4e70258"
server_key_iv: "2426154db2dc6d150abfc88e9a98b499"
server_key_version: "encryptionkey@0"
I20260620 07:20:42.995577 31450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:43.010303 31450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:43.010991 31450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:43.011199 31450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:43.018771 31450 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:32871
I20260620 07:20:43.018784 31516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:32871 every 8 connection(s)
I20260620 07:20:43.019760 31450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:43.021698 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 31450
I20260620 07:20:43.021831 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:43.022135 11758 external_mini_cluster.cc:1428] Setting key aab06eed2c4553c98fb4885ffecd2872
I20260620 07:20:43.023257 31517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:43.029881 31517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d: Bootstrap starting.
I20260620 07:20:43.032446 31517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:43.033259 31517 log.cc:824] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:43.034461 31520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.023627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.11.123.190:32871)
0620 07:20:43.024469 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:43.024494 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:43.024548 (+ 54us) server_negotiation.cc:407] Connection header received
0620 07:20:43.025823 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:43.025850 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:43.026412 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:43.026843 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:43.027334 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.029237 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.030461 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.031230 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.031443 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.032368 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:43.032388 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:43.032405 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:43.032885 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:43.032943 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:43.032965 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:43.033085 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:43.033277 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 07:20:43.033571 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":150,"threads_started":1}
I20260620 07:20:43.035141 31517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d: No bootstrap required, opened a new log
I20260620 07:20:43.038496 31517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc365de1c034457a981729bc7ea746d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 32871 } }
I20260620 07:20:43.038724 31517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:43.038798 31517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc365de1c034457a981729bc7ea746d, State: Initialized, Role: FOLLOWER
I20260620 07:20:43.039220 31517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc365de1c034457a981729bc7ea746d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 32871 } }
I20260620 07:20:43.039364 31517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:43.039439 31517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:43.039584 31517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:43.040346 31517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc365de1c034457a981729bc7ea746d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 32871 } }
I20260620 07:20:43.040620 31517 leader_election.cc:302] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fc365de1c034457a981729bc7ea746d; no voters:
I20260620 07:20:43.040848 31517 leader_election.cc:288] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:43.040970 31522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:43.041235 31522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [term 1 LEADER]: Becoming Leader. State: Replica: 0fc365de1c034457a981729bc7ea746d, State: Running, Role: LEADER
I20260620 07:20:43.041563 31522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc365de1c034457a981729bc7ea746d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 32871 } }
I20260620 07:20:43.042126 31517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:43.043347 31524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fc365de1c034457a981729bc7ea746d. Latest consensus state: current_term: 1 leader_uuid: "0fc365de1c034457a981729bc7ea746d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc365de1c034457a981729bc7ea746d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 32871 } } }
I20260620 07:20:43.043344 31523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fc365de1c034457a981729bc7ea746d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc365de1c034457a981729bc7ea746d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 32871 } } }
I20260620 07:20:43.043476 31524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:43.043494 31523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:43.043874 31531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:43.046924 31531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:43.052415 31531 catalog_manager.cc:1347] Generated new cluster ID: 0af7debea4f9435a9419761c76f0f108
I20260620 07:20:43.052490 31531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:43.071421 31531 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:43.072268 31531 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:43.079203 31531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d: Generated new TSK 0
I20260620 07:20:43.079821 31531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:43.093626 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.129:0
--local_ip_for_outbound_sockets=127.11.123.129
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:32871
--builtin_ntp_servers=127.11.123.148:39005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:20:43.206110 31541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:43.206348 31541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:43.206385 31541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:43.210278 31541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:43.210325 31541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:43.210376 31541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.129
I20260620 07:20:43.214932 31541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.123.129
--webserver_port=0
--tserver_master_addrs=127.11.123.190:32871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.129
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:43.215873 31541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:43.217233 31541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:43.226490 31547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:43.226512 31546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:43.226613 31541 server_base.cc:1034] running on GCE node
W20260620 07:20:43.226807 31549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:43.227161 31541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:43.227777 31541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:43.228935 31541 hybrid_clock.cc:648] HybridClock initialized: now 1781940043228901 us; error 45 us; skew 500 ppm
I20260620 07:20:43.230764 31541 webserver.cc:460] Webserver started at http://127.11.123.129:36199/ using document root <none> and password file <none>
I20260620 07:20:43.231511 31541 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:43.231613 31541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:43.231874 31541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:43.234787 31541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/data/instance:
uuid: "3b319ca084504b3ea88c03a88bbcb337"
format_stamp: "Formatted at 2026-06-20 07:20:43 on dist-test-slave-j663"
server_key: "c9d499efa91039c9cbd99c726540c3e2"
server_key_iv: "11f503df84f6430ba4e2c9517a1fd1fc"
server_key_version: "encryptionkey@0"
I20260620 07:20:43.235502 31541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance:
uuid: "3b319ca084504b3ea88c03a88bbcb337"
format_stamp: "Formatted at 2026-06-20 07:20:43 on dist-test-slave-j663"
server_key: "c9d499efa91039c9cbd99c726540c3e2"
server_key_iv: "11f503df84f6430ba4e2c9517a1fd1fc"
server_key_version: "encryptionkey@0"
I20260620 07:20:43.240401 31541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:20:43.243356 31555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:43.244762 31541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:20:43.244887 31541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/wal
uuid: "3b319ca084504b3ea88c03a88bbcb337"
format_stamp: "Formatted at 2026-06-20 07:20:43 on dist-test-slave-j663"
server_key: "c9d499efa91039c9cbd99c726540c3e2"
server_key_iv: "11f503df84f6430ba4e2c9517a1fd1fc"
server_key_version: "encryptionkey@0"
I20260620 07:20:43.244998 31541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:43.264451 31541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:43.265205 31541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:43.265399 31541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:43.265983 31541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:43.266901 31541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:43.266947 31541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:43.267015 31541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:43.267045 31541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:43.275807 31541 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.129:46147
I20260620 07:20:43.275838 31668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.129:46147 every 8 connection(s)
I20260620 07:20:43.276731 31541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/data/info.pb
I20260620 07:20:43.280216 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 31541
I20260620 07:20:43.280354 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-0/wal/instance
I20260620 07:20:43.280627 11758 external_mini_cluster.cc:1428] Setting key e3feb3c5833a13e3e1f3b6584f6ae9c8
I20260620 07:20:43.282713 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.130:0
--local_ip_for_outbound_sockets=127.11.123.130
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:32871
--builtin_ntp_servers=127.11.123.148:39005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:20:43.287055 31520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.278609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.129:39907 (local address 127.11.123.190:32871)
0620 07:20:43.278760 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:43.278765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:43.279847 (+ 1082us) server_negotiation.cc:407] Connection header received
0620 07:20:43.281428 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:43.281432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:43.281491 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:43.281582 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:43.282539 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.283356 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.284915 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.285195 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.285308 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.286412 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:43.286416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:43.286418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:43.286431 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:43.286458 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:43.286466 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:43.286563 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:43.286750 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 07:20:43.286834 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":53}
I20260620 07:20:43.287789 31671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.279045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:32871 (local address 127.11.123.129:39907)
0620 07:20:43.279690 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:43.279735 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:43.281177 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:43.281736 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:43.281742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:43.281827 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:43.282392 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.282407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.283490 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.283493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.284785 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.284798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.285358 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.285363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.285527 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.286151 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:20:43.286153 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:43.286193 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:20:43.286255 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:43.286549 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:43.286559 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:43.286565 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:20:43.286707 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 07:20:43.286962 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260620 07:20:43.289009 31669 heartbeater.cc:347] Connected to a master server at 127.11.123.190:32871
I20260620 07:20:43.289258 31669 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:43.289768 31669 heartbeater.cc:511] Master 127.11.123.190:32871 requested a full tablet report, sending...
I20260620 07:20:43.291218 31481 ts_manager.cc:194] Registered new tserver with Master: 3b319ca084504b3ea88c03a88bbcb337 (127.11.123.129:46147)
I20260620 07:20:43.292327 31481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.123.129:39907
W20260620 07:20:43.397022 31672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:43.397297 31672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:43.397336 31672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:43.401733 31672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:43.401790 31672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:43.401845 31672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.130
I20260620 07:20:43.407454 31672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.123.130
--webserver_port=0
--tserver_master_addrs=127.11.123.190:32871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.130
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:43.408619 31672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:43.410068 31672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:43.419499 31680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:43.419495 31677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:43.419502 31678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:43.419821 31672 server_base.cc:1034] running on GCE node
I20260620 07:20:43.420223 31672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:43.420887 31672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:43.422073 31672 hybrid_clock.cc:648] HybridClock initialized: now 1781940043422046 us; error 50 us; skew 500 ppm
I20260620 07:20:43.423967 31672 webserver.cc:460] Webserver started at http://127.11.123.130:46749/ using document root <none> and password file <none>
I20260620 07:20:43.424546 31672 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:43.424641 31672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:43.424875 31672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:43.426949 31672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/data/instance:
uuid: "ef19535f0ef84cf6ac32ff82261889d1"
format_stamp: "Formatted at 2026-06-20 07:20:43 on dist-test-slave-j663"
server_key: "a9523c01ba3c0ca3ae0a6a7458e598ce"
server_key_iv: "b0641bef2746792cbd4101d1d5822a39"
server_key_version: "encryptionkey@0"
I20260620 07:20:43.427515 31672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance:
uuid: "ef19535f0ef84cf6ac32ff82261889d1"
format_stamp: "Formatted at 2026-06-20 07:20:43 on dist-test-slave-j663"
server_key: "a9523c01ba3c0ca3ae0a6a7458e598ce"
server_key_iv: "b0641bef2746792cbd4101d1d5822a39"
server_key_version: "encryptionkey@0"
I20260620 07:20:43.431064 31672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:20:43.433449 31686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:43.434599 31672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:20:43.434732 31672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/wal
uuid: "ef19535f0ef84cf6ac32ff82261889d1"
format_stamp: "Formatted at 2026-06-20 07:20:43 on dist-test-slave-j663"
server_key: "a9523c01ba3c0ca3ae0a6a7458e598ce"
server_key_iv: "b0641bef2746792cbd4101d1d5822a39"
server_key_version: "encryptionkey@0"
I20260620 07:20:43.434842 31672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:43.460124 31672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:43.460819 31672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:43.461012 31672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:43.461645 31672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:43.462631 31672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:43.462700 31672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:43.462771 31672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:43.462815 31672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:43.471747 31672 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.130:45735
I20260620 07:20:43.471796 31799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.130:45735 every 8 connection(s)
I20260620 07:20:43.472621 31672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/data/info.pb
I20260620 07:20:43.479611 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 31672
I20260620 07:20:43.479765 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/wal/instance
I20260620 07:20:43.480046 11758 external_mini_cluster.cc:1428] Setting key 8378162b901626898420405e72cfb2e4
I20260620 07:20:43.481695 31520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.474355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.130:43065 (local address 127.11.123.190:32871)
0620 07:20:43.474485 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:43.474488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:43.475448 (+ 960us) server_negotiation.cc:407] Connection header received
0620 07:20:43.476919 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:43.476922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:43.476981 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:43.477104 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:43.477996 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.478526 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.479667 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.479994 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.480108 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.481214 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:43.481218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:43.481221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:43.481234 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:43.481261 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:43.481269 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:43.481346 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:43.481503 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 07:20:43.481561 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:20:43.482040 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.123.131:0
--local_ip_for_outbound_sockets=127.11.123.131
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:32871
--builtin_ntp_servers=127.11.123.148:39005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:20:43.482568 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.474674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:32871 (local address 127.11.123.130:43065)
0620 07:20:43.475301 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:43.475340 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:43.476703 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:43.477291 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:43.477297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:43.477378 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:43.477852 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.477864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.478658 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.478661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.479520 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.479543 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.480149 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.480153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.480323 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.480929 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:20:43.480930 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:43.480970 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:20:43.481028 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:43.481360 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:43.481369 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:43.481375 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:20:43.481503 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 07:20:43.481788 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260620 07:20:43.484236 31800 heartbeater.cc:347] Connected to a master server at 127.11.123.190:32871
I20260620 07:20:43.484467 31800 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:43.484953 31800 heartbeater.cc:511] Master 127.11.123.190:32871 requested a full tablet report, sending...
I20260620 07:20:43.486017 31481 ts_manager.cc:194] Registered new tserver with Master: ef19535f0ef84cf6ac32ff82261889d1 (127.11.123.130:45735)
I20260620 07:20:43.486552 31481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.123.130:43065
W20260620 07:20:43.592562 31803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:43.592830 31803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:43.592898 31803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:43.596870 31803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:43.596952 31803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:43.597043 31803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.123.131
I20260620 07:20:43.601985 31803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:39005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.123.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.123.131
--webserver_port=0
--tserver_master_addrs=127.11.123.190:32871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.123.131
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:43.603009 31803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:43.604346 31803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:43.613897 31808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:43.613929 31809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:43.613893 31811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:43.614307 31803 server_base.cc:1034] running on GCE node
I20260620 07:20:43.614753 31803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:43.615434 31803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:43.616640 31803 hybrid_clock.cc:648] HybridClock initialized: now 1781940043616612 us; error 47 us; skew 500 ppm
I20260620 07:20:43.618460 31803 webserver.cc:460] Webserver started at http://127.11.123.131:33765/ using document root <none> and password file <none>
I20260620 07:20:43.619076 31803 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:43.619139 31803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:43.619360 31803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:43.621500 31803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/data/instance:
uuid: "a8e3aa40eb1d4266b51575a695623626"
format_stamp: "Formatted at 2026-06-20 07:20:43 on dist-test-slave-j663"
server_key: "69295e08422fd340da8a3547ecae2a5d"
server_key_iv: "25c2b9bb6e5e75938f1deea6f1d78c7b"
server_key_version: "encryptionkey@0"
I20260620 07:20:43.622138 31803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance:
uuid: "a8e3aa40eb1d4266b51575a695623626"
format_stamp: "Formatted at 2026-06-20 07:20:43 on dist-test-slave-j663"
server_key: "69295e08422fd340da8a3547ecae2a5d"
server_key_iv: "25c2b9bb6e5e75938f1deea6f1d78c7b"
server_key_version: "encryptionkey@0"
I20260620 07:20:43.626089 31803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:20:43.628525 31817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:43.629899 31803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:20:43.630056 31803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/wal
uuid: "a8e3aa40eb1d4266b51575a695623626"
format_stamp: "Formatted at 2026-06-20 07:20:43 on dist-test-slave-j663"
server_key: "69295e08422fd340da8a3547ecae2a5d"
server_key_iv: "25c2b9bb6e5e75938f1deea6f1d78c7b"
server_key_version: "encryptionkey@0"
I20260620 07:20:43.630178 31803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:43.648420 31803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:43.649353 31803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:43.649536 31803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:43.650149 31803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:20:43.651394 31803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:20:43.651458 31803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:43.651528 31803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:20:43.651559 31803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:43.660475 31803 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.131:36197
I20260620 07:20:43.660498 31930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.131:36197 every 8 connection(s)
I20260620 07:20:43.661453 31803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/data/info.pb
I20260620 07:20:43.668195 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 31803
I20260620 07:20:43.668294 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/wal/instance
I20260620 07:20:43.668599 11758 external_mini_cluster.cc:1428] Setting key 430374226805f96af0a01f6dc6840077
I20260620 07:20:43.670868 31520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.663218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:34055 (local address 127.11.123.190:32871)
0620 07:20:43.663348 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:43.663354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:43.664338 (+ 984us) server_negotiation.cc:407] Connection header received
0620 07:20:43.665936 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:43.665939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:43.666000 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:43.666106 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:43.666989 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.667522 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.668786 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.669009 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.669166 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.670309 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:43.670312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:43.670314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:43.670324 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:43.670342 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:43.670349 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:43.670407 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:43.670679 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 07:20:43.670736 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:20:43.671657 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.663560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:32871 (local address 127.11.123.131:34055)
0620 07:20:43.664190 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:43.664233 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:43.665723 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:43.666277 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:43.666284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:43.666365 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:20:43.666842 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.666854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.667650 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.667660 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.668647 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.668662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.669176 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.669180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.669357 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.670036 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:20:43.670038 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:20:43.670089 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:20:43.670171 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:20:43.670432 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:20:43.670441 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:20:43.670446 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:20:43.670578 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 07:20:43.670855 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":113,"threads_started":1}
I20260620 07:20:43.672640 31931 heartbeater.cc:347] Connected to a master server at 127.11.123.190:32871
I20260620 07:20:43.672857 31931 heartbeater.cc:464] Registering TS with master...
I20260620 07:20:43.673394 31931 heartbeater.cc:511] Master 127.11.123.190:32871 requested a full tablet report, sending...
I20260620 07:20:43.674335 31481 ts_manager.cc:194] Registered new tserver with Master: a8e3aa40eb1d4266b51575a695623626 (127.11.123.131:36197)
I20260620 07:20:43.674831 31481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.123.131:34055
I20260620 07:20:43.682339 11758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:20:43.687902 31520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.684193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.11.123.190:32871)
0620 07:20:43.684325 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:43.684331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:43.684579 (+ 248us) server_negotiation.cc:407] Connection header received
0620 07:20:43.684675 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:43.684677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:43.684726 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:43.684788 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:43.685027 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.685647 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.686429 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.686625 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.686697 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.687471 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:43.687474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:43.687476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:43.687486 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:43.687502 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:43.687509 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:43.687587 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:43.687719 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 07:20:43.687769 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:20:43.690351 31481 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:20:43.693192 31481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:20:43.706923 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.700974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.131:36197 (local address 127.0.0.1:42666)
0620 07:20:43.701748 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:43.701794 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:43.701916 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:43.702599 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:43.702605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:43.702627 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:43.702773 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.702780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.703874 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.703880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.705172 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.705190 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.706431 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.706435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.706529 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.706552 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:20:43.706647 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 07:20:43.706771 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":111,"thread_start_us":123,"threads_started":1}
I20260620 07:20:43.707142 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.701718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.11.123.129:46147)
0620 07:20:43.702409 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:43.702423 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:43.702446 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:20:43.702521 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:43.702526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:43.702675 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:43.702836 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:43.703072 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.704274 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.705802 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.706545 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.706687 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.706841 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:43.706888 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 07:20:43.706988 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":139,"threads_started":1}
I20260620 07:20:43.707181 31943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.701286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:46147 (local address 127.0.0.1:35636)
0620 07:20:43.702041 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:43.702053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:43.702147 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:43.702814 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:43.702817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:43.702834 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:43.702939 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.702949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.704401 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.704405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.705660 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.705669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.706661 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.706666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.706752 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.706773 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:20:43.706835 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 07:20:43.706896 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":45,"thread_start_us":83,"threads_started":1}
I20260620 07:20:43.707407 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.701619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.11.123.131:36197)
0620 07:20:43.701997 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:43.702010 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:43.702034 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:20:43.702103 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:43.702110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:43.702318 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:43.702465 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:43.702974 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.703762 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.705312 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.706084 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.706204 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.706364 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:43.707110 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 07:20:43.707217 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":172,"threads_started":1}
I20260620 07:20:43.707453 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.700979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.11.123.130:45735)
0620 07:20:43.701699 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:43.701711 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:43.702250 (+ 539us) server_negotiation.cc:407] Connection header received
0620 07:20:43.702470 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:43.702475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:43.702645 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:43.702768 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:43.703163 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.704274 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.705753 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.706268 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.706402 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.706564 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:43.706734 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 07:20:43.706853 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":138,"threads_started":1}
I20260620 07:20:43.707881 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.700860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:45735 (local address 127.0.0.1:34210)
0620 07:20:43.701465 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:43.702153 (+ 688us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:43.702318 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:43.702940 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:43.702944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:43.702962 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:43.703046 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.703051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.704439 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.704441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.705615 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.705624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.706392 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.706396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.706498 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.706546 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 07:20:43.707679 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0620 07:20:43.707763 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":79,"thread_start_us":64,"threads_started":1}
I20260620 07:20:43.710088 31603 tablet_service.cc:1467] Processing CreateTablet for tablet 0510ebecd548456ab848b5c89476d4ad (DEFAULT_TABLE table=test-table [id=89088aa64078484c9842840a05e2ef36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:43.710464 31734 tablet_service.cc:1467] Processing CreateTablet for tablet 0510ebecd548456ab848b5c89476d4ad (DEFAULT_TABLE table=test-table [id=89088aa64078484c9842840a05e2ef36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:43.711126 31603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0510ebecd548456ab848b5c89476d4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:43.711400 31734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0510ebecd548456ab848b5c89476d4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:43.712349 31865 tablet_service.cc:1467] Processing CreateTablet for tablet 0510ebecd548456ab848b5c89476d4ad (DEFAULT_TABLE table=test-table [id=89088aa64078484c9842840a05e2ef36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:20:43.713289 31865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0510ebecd548456ab848b5c89476d4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:43.718991 31948 tablet_bootstrap.cc:492] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1: Bootstrap starting.
I20260620 07:20:43.719028 31949 tablet_bootstrap.cc:492] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337: Bootstrap starting.
I20260620 07:20:43.720657 31950 tablet_bootstrap.cc:492] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626: Bootstrap starting.
I20260620 07:20:43.721112 31948 tablet_bootstrap.cc:654] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:43.721120 31949 tablet_bootstrap.cc:654] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:43.721982 31948 log.cc:824] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:43.722239 31949 log.cc:824] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:43.722939 31950 tablet_bootstrap.cc:654] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:43.723685 31950 log.cc:824] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:43.724269 31949 tablet_bootstrap.cc:492] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337: No bootstrap required, opened a new log
I20260620 07:20:43.724319 31948 tablet_bootstrap.cc:492] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1: No bootstrap required, opened a new log
I20260620 07:20:43.724473 31949 ts_tablet_manager.cc:1397] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 07:20:43.724550 31948 ts_tablet_manager.cc:1397] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:20:43.725682 31950 tablet_bootstrap.cc:492] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626: No bootstrap required, opened a new log
I20260620 07:20:43.725864 31950 ts_tablet_manager.cc:1397] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 07:20:43.728003 31949 raft_consensus.cc:348] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef19535f0ef84cf6ac32ff82261889d1" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45735 } } peers { permanent_uuid: "a8e3aa40eb1d4266b51575a695623626" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36197 } } peers { permanent_uuid: "3b319ca084504b3ea88c03a88bbcb337" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 46147 } }
I20260620 07:20:43.728246 31949 raft_consensus.cc:374] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:43.728298 31949 raft_consensus.cc:729] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b319ca084504b3ea88c03a88bbcb337, State: Initialized, Role: FOLLOWER
I20260620 07:20:43.728721 31949 consensus_queue.cc:260] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef19535f0ef84cf6ac32ff82261889d1" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45735 } } peers { permanent_uuid: "a8e3aa40eb1d4266b51575a695623626" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36197 } } peers { permanent_uuid: "3b319ca084504b3ea88c03a88bbcb337" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 46147 } }
I20260620 07:20:43.728785 31948 raft_consensus.cc:348] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef19535f0ef84cf6ac32ff82261889d1" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45735 } } peers { permanent_uuid: "a8e3aa40eb1d4266b51575a695623626" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36197 } } peers { permanent_uuid: "3b319ca084504b3ea88c03a88bbcb337" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 46147 } }
I20260620 07:20:43.729028 31948 raft_consensus.cc:374] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:43.729112 31948 raft_consensus.cc:729] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef19535f0ef84cf6ac32ff82261889d1, State: Initialized, Role: FOLLOWER
I20260620 07:20:43.729476 31669 heartbeater.cc:503] Master 127.11.123.190:32871 was elected leader, sending a full tablet report...
I20260620 07:20:43.729526 31948 consensus_queue.cc:260] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef19535f0ef84cf6ac32ff82261889d1" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45735 } } peers { permanent_uuid: "a8e3aa40eb1d4266b51575a695623626" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36197 } } peers { permanent_uuid: "3b319ca084504b3ea88c03a88bbcb337" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 46147 } }
I20260620 07:20:43.729827 31950 raft_consensus.cc:348] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef19535f0ef84cf6ac32ff82261889d1" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45735 } } peers { permanent_uuid: "a8e3aa40eb1d4266b51575a695623626" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36197 } } peers { permanent_uuid: "3b319ca084504b3ea88c03a88bbcb337" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 46147 } }
I20260620 07:20:43.730060 31950 raft_consensus.cc:374] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:43.730118 31950 raft_consensus.cc:729] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8e3aa40eb1d4266b51575a695623626, State: Initialized, Role: FOLLOWER
I20260620 07:20:43.730207 31800 heartbeater.cc:503] Master 127.11.123.190:32871 was elected leader, sending a full tablet report...
I20260620 07:20:43.730515 31948 ts_tablet_manager.cc:1428] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260620 07:20:43.730513 31950 consensus_queue.cc:260] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef19535f0ef84cf6ac32ff82261889d1" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45735 } } peers { permanent_uuid: "a8e3aa40eb1d4266b51575a695623626" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36197 } } peers { permanent_uuid: "3b319ca084504b3ea88c03a88bbcb337" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 46147 } }
I20260620 07:20:43.731410 31950 ts_tablet_manager.cc:1428] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260620 07:20:43.731750 31931 heartbeater.cc:503] Master 127.11.123.190:32871 was elected leader, sending a full tablet report...
I20260620 07:20:43.732363 31949 ts_tablet_manager.cc:1428] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
W20260620 07:20:43.777987 31670 tablet.cc:2369] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:20:43.912431 31932 tablet.cc:2369] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:20:43.960638 31956 raft_consensus.cc:482] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:20:43.960855 31956 raft_consensus.cc:504] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef19535f0ef84cf6ac32ff82261889d1" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45735 } } peers { permanent_uuid: "a8e3aa40eb1d4266b51575a695623626" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36197 } } peers { permanent_uuid: "3b319ca084504b3ea88c03a88bbcb337" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 46147 } }
I20260620 07:20:43.961959 31956 leader_election.cc:288] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef19535f0ef84cf6ac32ff82261889d1 (127.11.123.130:45735), 3b319ca084504b3ea88c03a88bbcb337 (127.11.123.129:46147)
I20260620 07:20:43.966938 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.962542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:56013 (local address 127.11.123.129:46147)
0620 07:20:43.962721 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:43.962727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:43.962811 (+ 84us) server_negotiation.cc:407] Connection header received
0620 07:20:43.962909 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:43.962912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:43.962952 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:43.963013 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:43.963289 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.964001 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.965609 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.966198 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.966325 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.966373 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:43.966748 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 07:20:43.966815 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:20:43.966938 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.962351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.123.131:34001 (local address 127.11.123.130:45735)
0620 07:20:43.962479 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:43.962483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:43.962500 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:20:43.962556 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:43.962559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:43.962641 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:43.962734 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:20:43.963102 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.963896 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.965609 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.966198 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.966325 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.966373 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:43.966732 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 07:20:43.966806 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:20:43.966969 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.962229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.130:45735 (local address 127.11.123.131:34001)
0620 07:20:43.962341 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:43.962357 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:43.962453 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:43.962785 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:43.962788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:43.962814 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:43.962956 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.962961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.964008 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.964011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.965477 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.965489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.966332 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.966336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.966596 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.966611 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 07:20:43.966700 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 07:20:43.966811 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":168,"spinlock_wait_cycles":768}
I20260620 07:20:43.966976 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.962229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:46147 (local address 127.11.123.131:56013)
0620 07:20:43.962694 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:43.962708 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:20:43.962820 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:20:43.963046 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:20:43.963049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:20:43.963071 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:20:43.963171 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.963175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.964100 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.964103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.965478 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:20:43.965489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:20:43.966336 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:20:43.966339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:20:43.966559 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:43.966611 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 07:20:43.966694 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:20:43.966811 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":280,"spinlock_wait_cycles":2304,"thread_start_us":107,"threads_started":1}
I20260620 07:20:43.967790 31623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0510ebecd548456ab848b5c89476d4ad" candidate_uuid: "a8e3aa40eb1d4266b51575a695623626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b319ca084504b3ea88c03a88bbcb337" is_pre_election: true
I20260620 07:20:43.967795 31754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0510ebecd548456ab848b5c89476d4ad" candidate_uuid: "a8e3aa40eb1d4266b51575a695623626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef19535f0ef84cf6ac32ff82261889d1" is_pre_election: true
I20260620 07:20:43.968107 31623 raft_consensus.cc:2454] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8e3aa40eb1d4266b51575a695623626 in term 0.
I20260620 07:20:43.968107 31754 raft_consensus.cc:2454] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8e3aa40eb1d4266b51575a695623626 in term 0.
I20260620 07:20:43.968604 31818 leader_election.cc:302] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8e3aa40eb1d4266b51575a695623626, ef19535f0ef84cf6ac32ff82261889d1; no voters:
I20260620 07:20:43.968865 31956 raft_consensus.cc:2781] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:20:43.968945 31956 raft_consensus.cc:482] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:20:43.968974 31956 raft_consensus.cc:3037] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:43.970005 31956 raft_consensus.cc:504] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef19535f0ef84cf6ac32ff82261889d1" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45735 } } peers { permanent_uuid: "a8e3aa40eb1d4266b51575a695623626" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36197 } } peers { permanent_uuid: "3b319ca084504b3ea88c03a88bbcb337" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 46147 } }
I20260620 07:20:43.970346 31956 leader_election.cc:288] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [CANDIDATE]: Term 1 election: Requested vote from peers ef19535f0ef84cf6ac32ff82261889d1 (127.11.123.130:45735), 3b319ca084504b3ea88c03a88bbcb337 (127.11.123.129:46147)
I20260620 07:20:43.970700 31754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0510ebecd548456ab848b5c89476d4ad" candidate_uuid: "a8e3aa40eb1d4266b51575a695623626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef19535f0ef84cf6ac32ff82261889d1"
I20260620 07:20:43.970700 31623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0510ebecd548456ab848b5c89476d4ad" candidate_uuid: "a8e3aa40eb1d4266b51575a695623626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b319ca084504b3ea88c03a88bbcb337"
I20260620 07:20:43.970827 31754 raft_consensus.cc:3037] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:43.970827 31623 raft_consensus.cc:3037] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:43.972062 31754 raft_consensus.cc:2454] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8e3aa40eb1d4266b51575a695623626 in term 1.
I20260620 07:20:43.972062 31623 raft_consensus.cc:2454] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8e3aa40eb1d4266b51575a695623626 in term 1.
I20260620 07:20:43.972386 31821 leader_election.cc:302] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b319ca084504b3ea88c03a88bbcb337, a8e3aa40eb1d4266b51575a695623626; no voters:
I20260620 07:20:43.972558 31956 raft_consensus.cc:2781] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:43.972787 31956 raft_consensus.cc:686] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 1 LEADER]: Becoming Leader. State: Replica: a8e3aa40eb1d4266b51575a695623626, State: Running, Role: LEADER
I20260620 07:20:43.973138 31956 consensus_queue.cc:237] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef19535f0ef84cf6ac32ff82261889d1" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45735 } } peers { permanent_uuid: "a8e3aa40eb1d4266b51575a695623626" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36197 } } peers { permanent_uuid: "3b319ca084504b3ea88c03a88bbcb337" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 46147 } }
W20260620 07:20:43.973675 31801 tablet.cc:2369] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:20:43.976087 31481 catalog_manager.cc:5495] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8e3aa40eb1d4266b51575a695623626 (127.11.123.131). New cstate: current_term: 1 leader_uuid: "a8e3aa40eb1d4266b51575a695623626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef19535f0ef84cf6ac32ff82261889d1" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8e3aa40eb1d4266b51575a695623626" member_type: VOTER last_known_addr { host: "127.11.123.131" port: 36197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b319ca084504b3ea88c03a88bbcb337" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 46147 } health_report { overall_health: UNKNOWN } } }
I20260620 07:20:44.002350 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:43.998653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42678 (local address 127.11.123.131:36197)
0620 07:20:43.998788 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:43.998793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:43.998807 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:20:43.998999 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:43.999001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:43.999054 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:43.999160 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:20:43.999416 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:44.000009 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:44.000865 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:44.001129 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:44.001240 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:44.001552 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:20:44.001948 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:20:44.002081 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:44.002156 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 07:20:44.002207 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 07:20:44.007949 31754 raft_consensus.cc:1261] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1 [term 1 FOLLOWER]: Refusing update from remote peer a8e3aa40eb1d4266b51575a695623626: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:20:44.007972 31623 raft_consensus.cc:1261] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337 [term 1 FOLLOWER]: Refusing update from remote peer a8e3aa40eb1d4266b51575a695623626: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:20:44.008654 31956 consensus_queue.cc:1035] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b319ca084504b3ea88c03a88bbcb337" member_type: VOTER last_known_addr { host: "127.11.123.129" port: 46147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:20:44.008919 31958 consensus_queue.cc:1035] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef19535f0ef84cf6ac32ff82261889d1" member_type: VOTER last_known_addr { host: "127.11.123.130" port: 45735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:20:44.016824 31962 mvcc.cc:205] Tried to move back new op lower bound from 7298826420249686016 to 7298826420116279296. Current Snapshot: MvccSnapshot[applied={T|T < 7298826420249686016}]
I20260620 07:20:44.021231 31963 mvcc.cc:205] Tried to move back new op lower bound from 7298826420249686016 to 7298826420116279296. Current Snapshot: MvccSnapshot[applied={T|T < 7298826420249686016 or (T in {7298826420249686016})}]
I20260620 07:20:44.027423 31479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:20:44.027595 31479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:20:44.030165 31479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fc365de1c034457a981729bc7ea746d: Sending DeleteTablet for 3 replicas of tablet 0510ebecd548456ab848b5c89476d4ad
I20260620 07:20:44.030885 31734 tablet_service.cc:1514] Processing DeleteTablet for tablet 0510ebecd548456ab848b5c89476d4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:44 UTC) from {username='slave'} at 127.0.0.1:34210
I20260620 07:20:44.030885 31865 tablet_service.cc:1514] Processing DeleteTablet for tablet 0510ebecd548456ab848b5c89476d4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:44 UTC) from {username='slave'} at 127.0.0.1:42666
I20260620 07:20:44.031085 31603 tablet_service.cc:1514] Processing DeleteTablet for tablet 0510ebecd548456ab848b5c89476d4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:20:44 UTC) from {username='slave'} at 127.0.0.1:35636
I20260620 07:20:44.031549 31974 tablet_replica.cc:330] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337: stopping tablet replica
I20260620 07:20:44.031895 31974 raft_consensus.cc:2229] T 0510ebecd548456ab848b5c89476d4ad P 3b319ca084504b3ea88c03a88bbcb337 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:20:44.032187 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 31541
I20260620 07:20:44.035162 31972 tablet_replica.cc:330] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1: stopping tablet replica
I20260620 07:20:44.035444 31972 raft_consensus.cc:2229] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:20:44.035737 31972 raft_consensus.cc:2258] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:20:44.036130 31973 tablet_replica.cc:330] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626: stopping tablet replica
I20260620 07:20:44.036379 31973 raft_consensus.cc:2229] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:20:44.036687 31973 raft_consensus.cc:2258] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:20:44.038219 31972 ts_tablet_manager.cc:1905] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:20:44.038219 31973 ts_tablet_manager.cc:1905] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:20:44.041004 31972 ts_tablet_manager.cc:1918] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:20:44.041005 31973 ts_tablet_manager.cc:1918] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:20:44.041127 31972 log.cc:1196] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-1/wal/wals/0510ebecd548456ab848b5c89476d4ad
I20260620 07:20:44.041244 31973 log.cc:1196] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626: Deleting WAL directory at /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939794025788-11758-0/minicluster-data/ts-2/wal/wals/0510ebecd548456ab848b5c89476d4ad
I20260620 07:20:44.041486 31972 ts_tablet_manager.cc:1939] T 0510ebecd548456ab848b5c89476d4ad P ef19535f0ef84cf6ac32ff82261889d1: Deleting consensus metadata
I20260620 07:20:44.041666 31973 ts_tablet_manager.cc:1939] T 0510ebecd548456ab848b5c89476d4ad P a8e3aa40eb1d4266b51575a695623626: Deleting consensus metadata
W20260620 07:20:44.042573 31468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.123.129:46147 (error 108)
I20260620 07:20:44.042814 31466 catalog_manager.cc:4841] TS a8e3aa40eb1d4266b51575a695623626 (127.11.123.131:36197): tablet 0510ebecd548456ab848b5c89476d4ad (table test-table [id=89088aa64078484c9842840a05e2ef36]) successfully deleted
I20260620 07:20:44.043077 31465 catalog_manager.cc:4841] TS ef19535f0ef84cf6ac32ff82261889d1 (127.11.123.130:45735): tablet 0510ebecd548456ab848b5c89476d4ad (table test-table [id=89088aa64078484c9842840a05e2ef36]) successfully deleted
I20260620 07:20:44.043340 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 31672
I20260620 07:20:44.045167 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:44.044813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.129:46147 (local address 127.0.0.1:35642)
0620 07:20:44.044969 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 07:20:44.045047 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.129:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:20:44.045843 31468 catalog_manager.cc:4584] TS 3b319ca084504b3ea88c03a88bbcb337 (127.11.123.129:46147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0510ebecd548456ab848b5c89476d4ad: Network error: Client connection negotiation failed: client connection to 127.11.123.129:46147: connect: Connection refused (error 111)
I20260620 07:20:44.049825 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 31803
I20260620 07:20:44.054800 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 31450
2026-06-20T07:20:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1237 ms)
[----------] 6 tests from EncryptionPolicyTest (10401 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:44 dist-test-slave-j663 krb5kdc[31978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:44 dist-test-slave-j663 krb5kdc[31978](info): set up 2 sockets
Jun 20 07:20:44 dist-test-slave-j663 krb5kdc[31978](info): commencing operation
krb5kdc: starting...
W20260620 07:20:45.113338 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:45 dist-test-slave-j663 krb5kdc[31978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:20:45.237164 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:45.351218 31991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:45.351454 31991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:45.351493 31991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:45.355492 31991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:45.355543 31991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:45.355562 31991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:45.355583 31991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:45.355640 31991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:45.360675 31991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:45.361690 31991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:45.362653 31991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:45.371049 31997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:45.371196 31991 server_base.cc:1034] running on GCE node
W20260620 07:20:45.371070 31996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:45.371071 31999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:45.371876 31991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:45.372948 31991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:45.374143 31991 hybrid_clock.cc:648] HybridClock initialized: now 1781940045374135 us; error 30 us; skew 500 ppm
Jun 20 07:20:45 dist-test-slave-j663 krb5kdc[31978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:45.377310 31991 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:20:45.378154 31991 webserver.cc:460] Webserver started at http://127.0.0.1:40165/ using document root <none> and password file <none>
I20260620 07:20:45.378702 31991 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:45.378788 31991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:45.378983 31991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:45.380946 31991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "8182e37c2eb14816887dfded908771e4"
format_stamp: "Formatted at 2026-06-20 07:20:45 on dist-test-slave-j663"
server_key: "dfeb712b5548e756e8f80c07306ffc83"
server_key_iv: "ab35d26b8dfffb04cef44910e73d81b0"
server_key_version: "encryptionkey@0"
I20260620 07:20:45.381541 31991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "8182e37c2eb14816887dfded908771e4"
format_stamp: "Formatted at 2026-06-20 07:20:45 on dist-test-slave-j663"
server_key: "dfeb712b5548e756e8f80c07306ffc83"
server_key_iv: "ab35d26b8dfffb04cef44910e73d81b0"
server_key_version: "encryptionkey@0"
I20260620 07:20:45.385017 31991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:20:45.387374 32006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:45.388602 31991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:20:45.388734 31991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "8182e37c2eb14816887dfded908771e4"
format_stamp: "Formatted at 2026-06-20 07:20:45 on dist-test-slave-j663"
server_key: "dfeb712b5548e756e8f80c07306ffc83"
server_key_iv: "ab35d26b8dfffb04cef44910e73d81b0"
server_key_version: "encryptionkey@0"
I20260620 07:20:45.388837 31991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:45.410522 31991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:45.413938 31991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:45.414168 31991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:45.421217 31991 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44369
I20260620 07:20:45.421260 32058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44369 every 8 connection(s)
I20260620 07:20:45.422158 31991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:45.423014 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 31991
I20260620 07:20:45.423157 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:45.423427 11758 external_mini_cluster.cc:1428] Setting key f5c15b017f62cd7cc2d2262d1a45d6a9
I20260620 07:20:45.425257 32059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:45.430495 32059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4: Bootstrap starting.
Jun 20 07:20:45 dist-test-slave-j663 krb5kdc[31978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:20:45.432847 32059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:45.433584 32059 log.cc:824] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:45.435339 32059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4: No bootstrap required, opened a new log
I20260620 07:20:45.437428 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:45.424742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34715 (local address 127.0.0.1:44369)
0620 07:20:45.425245 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:45.425260 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:45.425322 (+ 62us) server_negotiation.cc:407] Connection header received
0620 07:20:45.425968 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:45.425984 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:45.426314 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:45.426657 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:45.427449 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:45.428423 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:45.429340 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:45.429803 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:45.429978 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:45.432701 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:45.432706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:45.432718 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:45.432744 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:45.434520 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:45.435041 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:45.435046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:45.435050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:45.435153 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:45.435515 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:45.435518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:45.435520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:45.435893 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:45.436116 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:45.436444 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 07:20:45.436649 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":156,"threads_started":1}
I20260620 07:20:45.439865 32059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8182e37c2eb14816887dfded908771e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44369 } }
I20260620 07:20:45.440110 32059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:45.440198 32059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8182e37c2eb14816887dfded908771e4, State: Initialized, Role: FOLLOWER
I20260620 07:20:45.440642 32059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8182e37c2eb14816887dfded908771e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44369 } }
I20260620 07:20:45.440809 32059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:45.440881 32059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:45.440971 32059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:45.441955 32059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8182e37c2eb14816887dfded908771e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44369 } }
I20260620 07:20:45.442189 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:20:45.442353 32059 leader_election.cc:302] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8182e37c2eb14816887dfded908771e4; no voters:
I20260620 07:20:45.442685 32059 leader_election.cc:288] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:45.442754 32064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:45.442962 32064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [term 1 LEADER]: Becoming Leader. State: Replica: 8182e37c2eb14816887dfded908771e4, State: Running, Role: LEADER
I20260620 07:20:45.443293 32064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8182e37c2eb14816887dfded908771e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44369 } }
I20260620 07:20:45.443878 32059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:45.444414 32064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8182e37c2eb14816887dfded908771e4. Latest consensus state: current_term: 1 leader_uuid: "8182e37c2eb14816887dfded908771e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8182e37c2eb14816887dfded908771e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44369 } } }
I20260620 07:20:45.444568 32064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:45.444820 32065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8182e37c2eb14816887dfded908771e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8182e37c2eb14816887dfded908771e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44369 } } }
I20260620 07:20:45.444933 32065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:45.445050 32077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:45.448230 32077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:45.453761 32077 catalog_manager.cc:1347] Generated new cluster ID: fa9dfa44b77f4ca7924a1bfa7e540c6a
I20260620 07:20:45.453851 32077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:45.454133 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:45.444305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58283 (local address 127.0.0.1:44369)
0620 07:20:45.445155 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:45.445160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:45.445174 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:20:45.445209 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:45.445211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:45.445251 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:45.446034 (+ 783us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:45.446179 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:45.446906 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:45.448010 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:45.448359 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:45.448826 (+ 467us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:45.450064 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:45.450070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:45.450075 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:45.450101 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:45.451802 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:45.452308 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:45.452315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:45.452317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:45.452407 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:45.452846 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:45.452849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:45.452850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:45.453045 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:45.453214 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:45.453956 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 07:20:45.454030 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737}
I20260620 07:20:45.467880 32077 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:45.468824 32077 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:45.476765 32077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8182e37c2eb14816887dfded908771e4: Generated new TSK 0
I20260620 07:20:45.477490 32077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:45.481259 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 31991
W20260620 07:20:45.486059 31976 connection.cc:535] client connection to 127.0.0.1:44369 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44369): Connection reset by peer (error 104)
2026-06-20T07:20:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:45 dist-test-slave-j663 krb5kdc[32090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:45 dist-test-slave-j663 krb5kdc[32090](info): set up 2 sockets
Jun 20 07:20:45 dist-test-slave-j663 krb5kdc[32090](info): commencing operation
krb5kdc: starting...
W20260620 07:20:46.573453 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:46 dist-test-slave-j663 krb5kdc[32090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:20:46.691715 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:46.798594 32103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:46.798889 32103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:46.798965 32103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:46.802944 32103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:46.803031 32103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:46.803076 32103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:46.803138 32103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:46.803174 32103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:46.808215 32103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37265
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:46.809226 32103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:46.810213 32103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:46.818516 32111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:46.818534 32109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:46.818601 32108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:46.818924 32103 server_base.cc:1034] running on GCE node
I20260620 07:20:46.819541 32103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:46.820748 32103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:46.821944 32103 hybrid_clock.cc:648] HybridClock initialized: now 1781940046821905 us; error 55 us; skew 500 ppm
Jun 20 07:20:46 dist-test-slave-j663 krb5kdc[32090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:46.824592 32103 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:20:46.825598 32103 webserver.cc:460] Webserver started at http://127.0.0.1:41127/ using document root <none> and password file <none>
I20260620 07:20:46.826162 32103 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:46.826258 32103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:46.826488 32103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:46.828467 32103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "1e8aad7ab3f843768af45a7bb365b43d"
format_stamp: "Formatted at 2026-06-20 07:20:46 on dist-test-slave-j663"
server_key: "a20e08916b915e324c4e932f5df8d98d"
server_key_iv: "8c7f48bf4ba9c1247bee13e70e49fe61"
server_key_version: "encryptionkey@0"
I20260620 07:20:46.829013 32103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "1e8aad7ab3f843768af45a7bb365b43d"
format_stamp: "Formatted at 2026-06-20 07:20:46 on dist-test-slave-j663"
server_key: "a20e08916b915e324c4e932f5df8d98d"
server_key_iv: "8c7f48bf4ba9c1247bee13e70e49fe61"
server_key_version: "encryptionkey@0"
I20260620 07:20:46.832626 32103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:20:46.835155 32118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:46.836423 32103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:20:46.836603 32103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "1e8aad7ab3f843768af45a7bb365b43d"
format_stamp: "Formatted at 2026-06-20 07:20:46 on dist-test-slave-j663"
server_key: "a20e08916b915e324c4e932f5df8d98d"
server_key_iv: "8c7f48bf4ba9c1247bee13e70e49fe61"
server_key_version: "encryptionkey@0"
I20260620 07:20:46.836728 32103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:46.855001 32103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:46.858009 32103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:46.858209 32103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:46.864961 32103 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37265
I20260620 07:20:46.865051 32170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37265 every 8 connection(s)
I20260620 07:20:46.865928 32103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:46.866900 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 32103
I20260620 07:20:46.867028 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:46.867342 11758 external_mini_cluster.cc:1428] Setting key 882422bb41bb74186664b90577d2f3a7
I20260620 07:20:46.868990 32171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:46.875564 32171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d: Bootstrap starting.
I20260620 07:20:46.877354 32174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:46.868686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53523 (local address 127.0.0.1:37265)
0620 07:20:46.869149 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:46.869168 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:46.869210 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:20:46.870003 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:46.870025 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:46.870446 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:46.870799 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:46.871193 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:46.872668 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:46.873696 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:46.874374 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:46.874539 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:46.875377 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:46.875383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:46.875397 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:46.875876 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:46.875950 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:46.875978 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:46.876105 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:46.876289 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 07:20:46.876410 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":113,"threads_started":1}
I20260620 07:20:46.878221 32171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:46.878969 32171 log.cc:824] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:46.880270 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:20:46.881147 32171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d: No bootstrap required, opened a new log
I20260620 07:20:46.885537 32171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8aad7ab3f843768af45a7bb365b43d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37265 } }
I20260620 07:20:46.885808 32171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:46.885883 32171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8aad7ab3f843768af45a7bb365b43d, State: Initialized, Role: FOLLOWER
I20260620 07:20:46.886364 32171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8aad7ab3f843768af45a7bb365b43d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37265 } }
I20260620 07:20:46.886525 32171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:46.886607 32171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:46.887040 32171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:46.887859 32174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:46.882484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44129 (local address 127.0.0.1:37265)
0620 07:20:46.883213 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:46.883217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:46.883239 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:20:46.883275 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:46.883278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:46.883325 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:46.883420 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:46.883766 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:46.884528 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:46.885784 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:46.886127 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:46.886235 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:46.887348 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:46.887354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:46.887357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:46.887375 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:46.887402 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:46.887415 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:46.887514 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:46.887691 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 07:20:46.887725 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634}
I20260620 07:20:46.888201 32171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8aad7ab3f843768af45a7bb365b43d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37265 } }
I20260620 07:20:46.888533 32171 leader_election.cc:302] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e8aad7ab3f843768af45a7bb365b43d; no voters:
I20260620 07:20:46.888856 32171 leader_election.cc:288] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:46.888926 32181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:46.889148 32181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [term 1 LEADER]: Becoming Leader. State: Replica: 1e8aad7ab3f843768af45a7bb365b43d, State: Running, Role: LEADER
I20260620 07:20:46.889539 32181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8aad7ab3f843768af45a7bb365b43d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37265 } }
I20260620 07:20:46.890159 32171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:46.890805 32181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e8aad7ab3f843768af45a7bb365b43d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8aad7ab3f843768af45a7bb365b43d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37265 } } }
I20260620 07:20:46.891022 32181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:46.891297 32181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e8aad7ab3f843768af45a7bb365b43d. Latest consensus state: current_term: 1 leader_uuid: "1e8aad7ab3f843768af45a7bb365b43d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8aad7ab3f843768af45a7bb365b43d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37265 } } }
I20260620 07:20:46.891443 32181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:46.891742 32190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:46.894517 32190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:46.899884 32190 catalog_manager.cc:1347] Generated new cluster ID: de318ba639ed454aaca96a40e75b5f31
I20260620 07:20:46.899952 32190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:46.945482 32190 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:46.946498 32190 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:46.956014 32190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e8aad7ab3f843768af45a7bb365b43d: Generated new TSK 0
I20260620 07:20:46.956759 32190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:46.966311 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 32103
W20260620 07:20:46.972112 32088 connection.cc:535] client connection to 127.0.0.1:37265 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37265): Connection reset by peer (error 104)
2026-06-20T07:20:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:47 dist-test-slave-j663 krb5kdc[32202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:47 dist-test-slave-j663 krb5kdc[32202](info): set up 2 sockets
Jun 20 07:20:47 dist-test-slave-j663 krb5kdc[32202](info): commencing operation
krb5kdc: starting...
W20260620 07:20:48.009392 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:48 dist-test-slave-j663 krb5kdc[32202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:20:48.125946 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:48.233852 32215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:48.234102 32215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:48.234143 32215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:48.239200 32215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:48.239267 32215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:48.239287 32215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:48.239305 32215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:48.239319 32215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:48.244526 32215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:48.245551 32215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:48.246502 32215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:48.254840 32221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:48.254863 32220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:48.254920 32215 server_base.cc:1034] running on GCE node
W20260620 07:20:48.254866 32223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:48.255561 32215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:48.256527 32215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:48.257682 32215 hybrid_clock.cc:648] HybridClock initialized: now 1781940048257663 us; error 30 us; skew 500 ppm
Jun 20 07:20:48 dist-test-slave-j663 krb5kdc[32202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:48.261261 32215 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:20:48.262096 32215 webserver.cc:460] Webserver started at http://127.0.0.1:44343/ using document root <none> and password file <none>
I20260620 07:20:48.262639 32215 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:48.262701 32215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:48.262921 32215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:48.264851 32215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "77e7907c6098475faa92d3bfc90e58e2"
format_stamp: "Formatted at 2026-06-20 07:20:48 on dist-test-slave-j663"
server_key: "c357bfe832690110e89148ade309172b"
server_key_iv: "03557a0441a18799ed5fd3f63479df2c"
server_key_version: "encryptionkey@0"
I20260620 07:20:48.265440 32215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "77e7907c6098475faa92d3bfc90e58e2"
format_stamp: "Formatted at 2026-06-20 07:20:48 on dist-test-slave-j663"
server_key: "c357bfe832690110e89148ade309172b"
server_key_iv: "03557a0441a18799ed5fd3f63479df2c"
server_key_version: "encryptionkey@0"
I20260620 07:20:48.269058 32215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:20:48.271775 32230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:48.272926 32215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:20:48.273022 32215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "77e7907c6098475faa92d3bfc90e58e2"
format_stamp: "Formatted at 2026-06-20 07:20:48 on dist-test-slave-j663"
server_key: "c357bfe832690110e89148ade309172b"
server_key_iv: "03557a0441a18799ed5fd3f63479df2c"
server_key_version: "encryptionkey@0"
I20260620 07:20:48.273181 32215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:48.277576 32215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:48.280651 32215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:48.280791 32215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:48.287580 32215 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38061
I20260620 07:20:48.287662 32282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38061 every 8 connection(s)
I20260620 07:20:48.288477 32215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:48.291181 32283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:48.291313 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 32215
I20260620 07:20:48.291468 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:48.291778 11758 external_mini_cluster.cc:1428] Setting key e97d95c218432b3ac2bb6287c9233d01
I20260620 07:20:48.296877 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:48.293057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60879 (local address 127.0.0.1:38061)
0620 07:20:48.293497 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:48.293515 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:48.293552 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:20:48.294373 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:48.294391 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:48.294664 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:48.295018 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:48.295769 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:48.295776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:48.295793 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:48.295803 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:48.295857 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:48.295875 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:48.296022 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:48.296092 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 07:20:48.296246 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":134,"threads_started":1}
I20260620 07:20:48.297698 32283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2: Bootstrap starting.
I20260620 07:20:48.299640 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:20:48.300060 32283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:48.300983 32283 log.cc:824] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:48.303231 32283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2: No bootstrap required, opened a new log
I20260620 07:20:48.303684 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:48.301830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57719 (local address 127.0.0.1:38061)
0620 07:20:48.302333 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:48.302337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:48.302350 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:20:48.302381 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:48.302384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:48.302411 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:48.302491 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:48.303287 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:48.303290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:48.303293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:48.303305 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:48.303331 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:48.303343 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:48.303423 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:48.303530 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 07:20:48.303572 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433}
I20260620 07:20:48.307104 32283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e7907c6098475faa92d3bfc90e58e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38061 } }
I20260620 07:20:48.307322 32283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:48.307381 32283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e7907c6098475faa92d3bfc90e58e2, State: Initialized, Role: FOLLOWER
I20260620 07:20:48.307821 32283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e7907c6098475faa92d3bfc90e58e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38061 } }
I20260620 07:20:48.307965 32283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:48.308035 32283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:48.308172 32283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:48.309502 32283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e7907c6098475faa92d3bfc90e58e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38061 } }
I20260620 07:20:48.309875 32283 leader_election.cc:302] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77e7907c6098475faa92d3bfc90e58e2; no voters:
I20260620 07:20:48.310200 32283 leader_election.cc:288] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:48.310304 32293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:48.310549 32293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [term 1 LEADER]: Becoming Leader. State: Replica: 77e7907c6098475faa92d3bfc90e58e2, State: Running, Role: LEADER
I20260620 07:20:48.310885 32293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e7907c6098475faa92d3bfc90e58e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38061 } }
I20260620 07:20:48.311395 32283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:48.312493 32295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77e7907c6098475faa92d3bfc90e58e2. Latest consensus state: current_term: 1 leader_uuid: "77e7907c6098475faa92d3bfc90e58e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e7907c6098475faa92d3bfc90e58e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38061 } } }
I20260620 07:20:48.312619 32295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:48.312580 32294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77e7907c6098475faa92d3bfc90e58e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e7907c6098475faa92d3bfc90e58e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38061 } } }
I20260620 07:20:48.312669 32294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:48.313016 32302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:48.316174 32302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:48.321452 32302 catalog_manager.cc:1347] Generated new cluster ID: 35d8953d73e446379d4286757f64c6f6
I20260620 07:20:48.321514 32302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:48.337009 32302 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:48.337342 32302 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:48.347883 32302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77e7907c6098475faa92d3bfc90e58e2: Generated new TSK 0
I20260620 07:20:48.348752 32302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:48.352058 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 32215
2026-06-20T07:20:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:48 dist-test-slave-j663 krb5kdc[32314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:48 dist-test-slave-j663 krb5kdc[32314](info): set up 2 sockets
Jun 20 07:20:48 dist-test-slave-j663 krb5kdc[32314](info): commencing operation
krb5kdc: starting...
W20260620 07:20:49.397361 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:49 dist-test-slave-j663 krb5kdc[32314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:20:49.522090 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:49.630633 32327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:49.631080 32327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:49.631197 32327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:49.635674 32327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:49.635741 32327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:49.635762 32327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:49.635779 32327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:49.635795 32327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:49.641592 32327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:49.642590 32327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:49.643525 32327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:49.651749 32333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:49.651842 32327 server_base.cc:1034] running on GCE node
W20260620 07:20:49.651921 32332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:49.651862 32335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:49.652716 32327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:49.654160 32327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:49.655339 32327 hybrid_clock.cc:648] HybridClock initialized: now 1781940049655319 us; error 31 us; skew 500 ppm
Jun 20 07:20:49 dist-test-slave-j663 krb5kdc[32314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:49.661304 32327 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:20:49.662109 32327 webserver.cc:460] Webserver started at http://127.0.0.1:43893/ using document root <none> and password file <none>
I20260620 07:20:49.662621 32327 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:49.662680 32327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:49.662853 32327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:49.664786 32327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "f258f7ee2199400b9d9c27ca97e3f8f5"
format_stamp: "Formatted at 2026-06-20 07:20:49 on dist-test-slave-j663"
server_key: "1e21564e9fcbaa5689e5ff7582575abb"
server_key_iv: "a23f52a77b3cb8c8f1b9abef095b7216"
server_key_version: "encryptionkey@0"
I20260620 07:20:49.665370 32327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "f258f7ee2199400b9d9c27ca97e3f8f5"
format_stamp: "Formatted at 2026-06-20 07:20:49 on dist-test-slave-j663"
server_key: "1e21564e9fcbaa5689e5ff7582575abb"
server_key_iv: "a23f52a77b3cb8c8f1b9abef095b7216"
server_key_version: "encryptionkey@0"
I20260620 07:20:49.668818 32327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:20:49.671133 32342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:49.672308 32327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:20:49.672474 32327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "f258f7ee2199400b9d9c27ca97e3f8f5"
format_stamp: "Formatted at 2026-06-20 07:20:49 on dist-test-slave-j663"
server_key: "1e21564e9fcbaa5689e5ff7582575abb"
server_key_iv: "a23f52a77b3cb8c8f1b9abef095b7216"
server_key_version: "encryptionkey@0"
I20260620 07:20:49.672581 32327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:49.698519 32327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:49.701776 32327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:49.701975 32327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:49.708935 32327 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35171
I20260620 07:20:49.708945 32394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35171 every 8 connection(s)
I20260620 07:20:49.709951 32327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:49.713225 32395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:49.718005 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 32327
I20260620 07:20:49.718146 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:49.718439 11758 external_mini_cluster.cc:1428] Setting key 340b7c64b5e1807ca3cfd55fa87d7091
I20260620 07:20:49.718868 32395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5: Bootstrap starting.
I20260620 07:20:49.721329 32395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:49.722174 32395 log.cc:824] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:49.724118 32395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5: No bootstrap required, opened a new log
I20260620 07:20:49.728312 32395 raft_consensus.cc:348] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f258f7ee2199400b9d9c27ca97e3f8f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35171 } }
I20260620 07:20:49.728591 32395 raft_consensus.cc:374] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:49.728695 32395 raft_consensus.cc:729] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f258f7ee2199400b9d9c27ca97e3f8f5, State: Initialized, Role: FOLLOWER
I20260620 07:20:49.729238 32395 consensus_queue.cc:260] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f258f7ee2199400b9d9c27ca97e3f8f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35171 } }
I20260620 07:20:49.729439 32395 raft_consensus.cc:388] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:49.729517 32395 raft_consensus.cc:482] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:49.729630 32395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 07:20:49 dist-test-slave-j663 krb5kdc[32314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:20:49.730688 32395 raft_consensus.cc:504] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f258f7ee2199400b9d9c27ca97e3f8f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35171 } }
I20260620 07:20:49.731029 32395 leader_election.cc:302] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f258f7ee2199400b9d9c27ca97e3f8f5; no voters:
I20260620 07:20:49.731415 32395 leader_election.cc:288] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:49.731518 32400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:49.731750 32400 raft_consensus.cc:686] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [term 1 LEADER]: Becoming Leader. State: Replica: f258f7ee2199400b9d9c27ca97e3f8f5, State: Running, Role: LEADER
I20260620 07:20:49.732151 32400 consensus_queue.cc:237] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f258f7ee2199400b9d9c27ca97e3f8f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35171 } }
I20260620 07:20:49.732748 32395 sys_catalog.cc:564] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:49.733891 32401 sys_catalog.cc:455] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f258f7ee2199400b9d9c27ca97e3f8f5. Latest consensus state: current_term: 1 leader_uuid: "f258f7ee2199400b9d9c27ca97e3f8f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f258f7ee2199400b9d9c27ca97e3f8f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35171 } } }
I20260620 07:20:49.734031 32401 sys_catalog.cc:458] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:49.734452 32402 sys_catalog.cc:455] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f258f7ee2199400b9d9c27ca97e3f8f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f258f7ee2199400b9d9c27ca97e3f8f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35171 } } }
I20260620 07:20:49.734594 32402 sys_catalog.cc:458] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:49.735354 32409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:49.736861 32398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:49.719790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47175 (local address 127.0.0.1:35171)
0620 07:20:49.720274 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:49.720289 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:49.720324 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:20:49.720924 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:49.720939 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:49.721470 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:49.721812 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:49.724286 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:49.725787 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:49.727096 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:49.727812 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:49.727976 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:49.730925 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:49.730930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:49.730942 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:49.730968 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:49.733118 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:49.733623 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:49.733630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:49.733636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:49.733796 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:49.734154 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:49.734157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:49.734159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:49.734701 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:49.734882 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:49.735868 (+ 986us) server_negotiation.cc:299] Negotiation successful
0620 07:20:49.736155 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":156,"threads_started":1}
I20260620 07:20:49.738498 32409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:49.739675 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:20:49.744515 32409 catalog_manager.cc:1347] Generated new cluster ID: c8f34887b72e4a89a7842bd3f1e3694d
I20260620 07:20:49.744588 32409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:49.750633 32398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:49.741736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58931 (local address 127.0.0.1:35171)
0620 07:20:49.741846 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:49.741851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:49.742111 (+ 260us) server_negotiation.cc:407] Connection header received
0620 07:20:49.742226 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:49.742229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:49.742277 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:49.742365 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:49.743212 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:49.743941 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:49.745753 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:49.746007 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:49.746129 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:49.747454 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:49.747457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:49.747460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:49.747484 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:49.748949 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:49.749414 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:49.749418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:49.749420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:49.749499 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:49.749794 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:49.749799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:49.749800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:49.749980 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:49.750138 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:49.750351 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 07:20:49.750466 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 07:20:49.764647 32409 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:49.765965 32409 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:49.776582 32409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f258f7ee2199400b9d9c27ca97e3f8f5: Generated new TSK 0
I20260620 07:20:49.777380 32409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:49.784011 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 32327
2026-06-20T07:20:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:49 dist-test-slave-j663 krb5kdc[32426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:49 dist-test-slave-j663 krb5kdc[32426](info): set up 2 sockets
Jun 20 07:20:49 dist-test-slave-j663 krb5kdc[32426](info): commencing operation
krb5kdc: starting...
W20260620 07:20:50.829361 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:50 dist-test-slave-j663 krb5kdc[32426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:20:50.951625 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43869
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:51.062458 32439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:51.062695 32439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:51.062736 32439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:51.066677 32439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:51.066738 32439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:51.066761 32439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:51.066778 32439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:51.066793 32439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:51.071944 32439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:51.072840 32439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:51.073781 32439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:51.081977 32444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:51.082062 32439 server_base.cc:1034] running on GCE node
W20260620 07:20:51.081955 32445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:51.082265 32447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:51.082782 32439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:51.083812 32439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:51.085000 32439 hybrid_clock.cc:648] HybridClock initialized: now 1781940051084979 us; error 30 us; skew 500 ppm
Jun 20 07:20:51 dist-test-slave-j663 krb5kdc[32426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:51.088426 32439 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:20:51.089640 32439 webserver.cc:460] Webserver started at http://127.0.0.1:43405/ using document root <none> and password file <none>
I20260620 07:20:51.090197 32439 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:51.090260 32439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:51.090417 32439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:51.092339 32439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "3e8cfba05b77446c92a5c24d7bfcac76"
format_stamp: "Formatted at 2026-06-20 07:20:51 on dist-test-slave-j663"
server_key: "8117ac375e7875e4990af33d16c493f1"
server_key_iv: "049aeaa5bdd0123c23b4433d739d6693"
server_key_version: "encryptionkey@0"
I20260620 07:20:51.092839 32439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "3e8cfba05b77446c92a5c24d7bfcac76"
format_stamp: "Formatted at 2026-06-20 07:20:51 on dist-test-slave-j663"
server_key: "8117ac375e7875e4990af33d16c493f1"
server_key_iv: "049aeaa5bdd0123c23b4433d739d6693"
server_key_version: "encryptionkey@0"
I20260620 07:20:51.096386 32439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:20:51.098731 32454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:51.100028 32439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:20:51.100127 32439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "3e8cfba05b77446c92a5c24d7bfcac76"
format_stamp: "Formatted at 2026-06-20 07:20:51 on dist-test-slave-j663"
server_key: "8117ac375e7875e4990af33d16c493f1"
server_key_iv: "049aeaa5bdd0123c23b4433d739d6693"
server_key_version: "encryptionkey@0"
I20260620 07:20:51.100212 32439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:51.115860 32439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:51.118636 32439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:51.118811 32439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:51.125705 32439 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43869
I20260620 07:20:51.125777 32506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43869 every 8 connection(s)
I20260620 07:20:51.126626 32439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:51.127441 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 32439
I20260620 07:20:51.127576 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:51.127871 11758 external_mini_cluster.cc:1428] Setting key ab3d861d74525fceb320d9173ceeb9db
I20260620 07:20:51.129925 32507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:51.135357 32507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76: Bootstrap starting.
I20260620 07:20:51.136567 32510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:51.129206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40721 (local address 127.0.0.1:43869)
0620 07:20:51.129643 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:51.129657 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:51.129691 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:20:51.130405 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:51.130423 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:51.130777 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:51.131113 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:51.131501 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:51.132459 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:51.133424 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:51.133883 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:51.134049 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:51.134923 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:51.134928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:51.134940 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:51.135339 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:51.135391 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:51.135438 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:51.135533 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:51.135712 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 07:20:51.135915 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":147,"threads_started":1}
I20260620 07:20:51.137790 32507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:51.138633 32507 log.cc:824] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:51.139436 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:20:51.140909 32507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76: No bootstrap required, opened a new log
I20260620 07:20:51.145632 32510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:51.141649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32933 (local address 127.0.0.1:43869)
0620 07:20:51.141774 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:51.141778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:51.141956 (+ 178us) server_negotiation.cc:407] Connection header received
0620 07:20:51.142065 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:51.142067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:51.142102 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:51.142166 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:51.142434 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:51.142909 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:51.144069 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:51.144276 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:51.144348 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:51.145144 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:51.145148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:51.145150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:51.145163 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:51.145192 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:51.145202 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:51.145269 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:51.145385 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 07:20:51.145428 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:20:51.145529 32507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8cfba05b77446c92a5c24d7bfcac76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43869 } }
I20260620 07:20:51.145818 32507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:51.145874 32507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8cfba05b77446c92a5c24d7bfcac76, State: Initialized, Role: FOLLOWER
I20260620 07:20:51.146330 32507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8cfba05b77446c92a5c24d7bfcac76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43869 } }
I20260620 07:20:51.146520 32507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:51.146615 32507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:51.146713 32507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:51.147563 32507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8cfba05b77446c92a5c24d7bfcac76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43869 } }
I20260620 07:20:51.147889 32507 leader_election.cc:302] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e8cfba05b77446c92a5c24d7bfcac76; no voters:
I20260620 07:20:51.148180 32507 leader_election.cc:288] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:51.148267 32517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:51.148511 32517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [term 1 LEADER]: Becoming Leader. State: Replica: 3e8cfba05b77446c92a5c24d7bfcac76, State: Running, Role: LEADER
I20260620 07:20:51.148851 32517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8cfba05b77446c92a5c24d7bfcac76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43869 } }
I20260620 07:20:51.149416 32507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:51.150506 32519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e8cfba05b77446c92a5c24d7bfcac76. Latest consensus state: current_term: 1 leader_uuid: "3e8cfba05b77446c92a5c24d7bfcac76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8cfba05b77446c92a5c24d7bfcac76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43869 } } }
I20260620 07:20:51.150643 32519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:51.151048 32526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:51.151007 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e8cfba05b77446c92a5c24d7bfcac76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8cfba05b77446c92a5c24d7bfcac76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43869 } } }
I20260620 07:20:51.151217 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:51.154716 32526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:51.160740 32526 catalog_manager.cc:1347] Generated new cluster ID: 1f713bee34c2417ca645624bb731538c
I20260620 07:20:51.160823 32526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:51.170133 32526 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:51.171077 32526 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:51.178213 32526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e8cfba05b77446c92a5c24d7bfcac76: Generated new TSK 0
I20260620 07:20:51.178823 32526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:51.187821 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 32439
2026-06-20T07:20:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:51 dist-test-slave-j663 krb5kdc[32538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:51 dist-test-slave-j663 krb5kdc[32538](info): set up 2 sockets
Jun 20 07:20:51 dist-test-slave-j663 krb5kdc[32538](info): commencing operation
krb5kdc: starting...
W20260620 07:20:52.241406 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:52 dist-test-slave-j663 krb5kdc[32538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:20:52.360347 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:52.472975 32551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:52.473222 32551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:52.473261 32551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:52.477188 32551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:52.477236 32551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:52.477254 32551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:52.477270 32551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:52.477285 32551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:52.482316 32551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:52.483287 32551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:52.484264 32551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:20:52.492616 32551 server_base.cc:1034] running on GCE node
W20260620 07:20:52.492586 32557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:52.492573 32559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:52.492550 32556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:52.493422 32551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:52.494390 32551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:52.495548 32551 hybrid_clock.cc:648] HybridClock initialized: now 1781940052495529 us; error 29 us; skew 500 ppm
Jun 20 07:20:52 dist-test-slave-j663 krb5kdc[32538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:52.498199 32551 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:20:52.499056 32551 webserver.cc:460] Webserver started at http://127.0.0.1:40215/ using document root <none> and password file <none>
I20260620 07:20:52.499584 32551 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:52.499697 32551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:52.499924 32551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:52.501909 32551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "28cc5d116f9f473abd1ddeea660768fe"
format_stamp: "Formatted at 2026-06-20 07:20:52 on dist-test-slave-j663"
server_key: "012951c34489a028196d32f0242423a2"
server_key_iv: "2a4e5c46ea5a5786ace976877bccfb69"
server_key_version: "encryptionkey@0"
I20260620 07:20:52.502450 32551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "28cc5d116f9f473abd1ddeea660768fe"
format_stamp: "Formatted at 2026-06-20 07:20:52 on dist-test-slave-j663"
server_key: "012951c34489a028196d32f0242423a2"
server_key_iv: "2a4e5c46ea5a5786ace976877bccfb69"
server_key_version: "encryptionkey@0"
I20260620 07:20:52.505949 32551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:20:52.508263 32566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:52.509517 32551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:20:52.509658 32551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "28cc5d116f9f473abd1ddeea660768fe"
format_stamp: "Formatted at 2026-06-20 07:20:52 on dist-test-slave-j663"
server_key: "012951c34489a028196d32f0242423a2"
server_key_iv: "2a4e5c46ea5a5786ace976877bccfb69"
server_key_version: "encryptionkey@0"
I20260620 07:20:52.509766 32551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:52.515692 32551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:52.518524 32551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:52.518672 32551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:52.525475 32551 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39073
I20260620 07:20:52.525498 32618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39073 every 8 connection(s)
I20260620 07:20:52.526347 32551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:52.528991 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:52.534333 32619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe: Bootstrap starting.
I20260620 07:20:52.535738 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 32551
I20260620 07:20:52.535873 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:52.536151 11758 external_mini_cluster.cc:1428] Setting key 2b037be96ea38a02334718da0e0e0988
I20260620 07:20:52.536970 32619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:52.537853 32619 log.cc:824] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:52.540083 32619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe: No bootstrap required, opened a new log
I20260620 07:20:52.541543 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:52.537304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49715 (local address 127.0.0.1:39073)
0620 07:20:52.537784 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:52.537802 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:52.537843 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:20:52.538639 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:52.538664 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:52.539004 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:52.539368 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:52.540097 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:52.540105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:52.540122 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:52.540137 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:52.540201 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:52.540228 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:52.540342 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:52.540474 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 07:20:52.540616 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":114,"threads_started":1}
I20260620 07:20:52.544420 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:20:52.544370 32619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28cc5d116f9f473abd1ddeea660768fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39073 } }
I20260620 07:20:52.544641 32619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:52.544715 32619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28cc5d116f9f473abd1ddeea660768fe, State: Initialized, Role: FOLLOWER
I20260620 07:20:52.545256 32619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28cc5d116f9f473abd1ddeea660768fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39073 } }
I20260620 07:20:52.545466 32619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:52.545552 32619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:52.545676 32619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:52.546658 32619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28cc5d116f9f473abd1ddeea660768fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39073 } }
I20260620 07:20:52.547116 32619 leader_election.cc:302] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28cc5d116f9f473abd1ddeea660768fe; no voters:
I20260620 07:20:52.547463 32619 leader_election.cc:288] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:52.547530 32629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:52.547756 32629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [term 1 LEADER]: Becoming Leader. State: Replica: 28cc5d116f9f473abd1ddeea660768fe, State: Running, Role: LEADER
I20260620 07:20:52.548099 32629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28cc5d116f9f473abd1ddeea660768fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39073 } }
I20260620 07:20:52.548802 32619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:52.548952 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:52.546908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55493 (local address 127.0.0.1:39073)
0620 07:20:52.547025 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:52.547029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:52.547197 (+ 168us) server_negotiation.cc:407] Connection header received
0620 07:20:52.547388 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:52.547394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:52.547439 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:52.547514 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:52.548535 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:52.548538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:52.548541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:52.548553 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:52.548587 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:52.548598 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:52.548690 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:52.548775 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 07:20:52.548816 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:20:52.549374 32629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28cc5d116f9f473abd1ddeea660768fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28cc5d116f9f473abd1ddeea660768fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39073 } } }
I20260620 07:20:52.549552 32629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:52.550002 32635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:52.549978 32630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28cc5d116f9f473abd1ddeea660768fe. Latest consensus state: current_term: 1 leader_uuid: "28cc5d116f9f473abd1ddeea660768fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28cc5d116f9f473abd1ddeea660768fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39073 } } }
I20260620 07:20:52.550165 32630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:52.553318 32635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:52.558616 32635 catalog_manager.cc:1347] Generated new cluster ID: 8811c2eb06a341c5aa3b39f58035a7a6
I20260620 07:20:52.558678 32635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:52.589927 32635 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:52.590281 32635 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:52.597537 32635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28cc5d116f9f473abd1ddeea660768fe: Generated new TSK 0
I20260620 07:20:52.598304 32635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:52.607609 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 32551
2026-06-20T07:20:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:52 dist-test-slave-j663 krb5kdc[32650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:52 dist-test-slave-j663 krb5kdc[32650](info): set up 2 sockets
Jun 20 07:20:52 dist-test-slave-j663 krb5kdc[32650](info): commencing operation
krb5kdc: starting...
W20260620 07:20:53.681452 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:53 dist-test-slave-j663 krb5kdc[32650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:20:53.799376 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33395
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:53.906173 32663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:53.906409 32663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:53.906448 32663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:53.910828 32663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:53.910882 32663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:53.910902 32663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:53.910918 32663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:53.910933 32663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:53.915931 32663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:53.916944 32663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:53.917991 32663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:53.926194 32671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:53.926189 32669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:53.926381 32668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:53.926280 32663 server_base.cc:1034] running on GCE node
I20260620 07:20:53.927094 32663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:53.928122 32663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:53.929302 32663 hybrid_clock.cc:648] HybridClock initialized: now 1781940053929290 us; error 23 us; skew 500 ppm
Jun 20 07:20:53 dist-test-slave-j663 krb5kdc[32650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:53.932164 32663 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:20:53.933207 32663 webserver.cc:460] Webserver started at http://127.0.0.1:35393/ using document root <none> and password file <none>
I20260620 07:20:53.933776 32663 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:53.933868 32663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:53.934147 32663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:53.936156 32663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "445a2b561b874807af31f6d6dc32539e"
format_stamp: "Formatted at 2026-06-20 07:20:53 on dist-test-slave-j663"
server_key: "9b062586645ea646ff93ed245efb302b"
server_key_iv: "4e58e5df7e23804dbaab09187b989097"
server_key_version: "encryptionkey@0"
I20260620 07:20:53.936726 32663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "445a2b561b874807af31f6d6dc32539e"
format_stamp: "Formatted at 2026-06-20 07:20:53 on dist-test-slave-j663"
server_key: "9b062586645ea646ff93ed245efb302b"
server_key_iv: "4e58e5df7e23804dbaab09187b989097"
server_key_version: "encryptionkey@0"
I20260620 07:20:53.940311 32663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 07:20:53.942662 32678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:53.943790 32663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:20:53.943925 32663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "445a2b561b874807af31f6d6dc32539e"
format_stamp: "Formatted at 2026-06-20 07:20:53 on dist-test-slave-j663"
server_key: "9b062586645ea646ff93ed245efb302b"
server_key_iv: "4e58e5df7e23804dbaab09187b989097"
server_key_version: "encryptionkey@0"
I20260620 07:20:53.944031 32663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:53.957070 32663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:53.960278 32663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:53.960475 32663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:53.967607 32663 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33395
I20260620 07:20:53.967629 32730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33395 every 8 connection(s)
I20260620 07:20:53.968581 32663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:53.971299 32731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:53.974843 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 32663
I20260620 07:20:53.974974 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:53.975229 11758 external_mini_cluster.cc:1428] Setting key b12c0fac4e748c6cd5b9c70e74d11a01
I20260620 07:20:53.976878 32731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e: Bootstrap starting.
I20260620 07:20:53.979020 32731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:53.979763 32731 log.cc:824] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:53.981590 32731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e: No bootstrap required, opened a new log
Jun 20 07:20:53 dist-test-slave-j663 krb5kdc[32650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:20:53.985507 32731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445a2b561b874807af31f6d6dc32539e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33395 } }
I20260620 07:20:53.985715 32731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:53.985792 32731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445a2b561b874807af31f6d6dc32539e, State: Initialized, Role: FOLLOWER
I20260620 07:20:53.986196 32731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445a2b561b874807af31f6d6dc32539e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33395 } }
I20260620 07:20:53.986315 32731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:53.986400 32731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:53.986512 32731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:53.987658 32731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445a2b561b874807af31f6d6dc32539e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33395 } }
I20260620 07:20:53.988049 32731 leader_election.cc:302] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 445a2b561b874807af31f6d6dc32539e; no voters:
I20260620 07:20:53.988396 32731 leader_election.cc:288] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:53.988489 32736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:53.988744 32736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [term 1 LEADER]: Becoming Leader. State: Replica: 445a2b561b874807af31f6d6dc32539e, State: Running, Role: LEADER
I20260620 07:20:53.989270 32736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445a2b561b874807af31f6d6dc32539e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33395 } }
I20260620 07:20:53.989734 32731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:53.990255 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:53.976568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:49367 (local address 127.0.0.1:33395)
0620 07:20:53.977003 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:53.977021 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:53.977065 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:20:53.977723 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:53.977739 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:53.978074 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:53.978411 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:53.979173 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:53.980225 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:53.981071 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:53.981559 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:53.981736 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:53.984531 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:53.984536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:53.984548 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:53.984573 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:53.986647 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:53.987179 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:53.987184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:53.987188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:53.987291 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:53.987658 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:53.987660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:53.987662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:53.988199 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:53.988423 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:53.989057 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 07:20:53.989321 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":131,"threads_started":1}
I20260620 07:20:53.990882 32738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 445a2b561b874807af31f6d6dc32539e. Latest consensus state: current_term: 1 leader_uuid: "445a2b561b874807af31f6d6dc32539e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445a2b561b874807af31f6d6dc32539e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33395 } } }
I20260620 07:20:53.990836 32737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "445a2b561b874807af31f6d6dc32539e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445a2b561b874807af31f6d6dc32539e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33395 } } }
I20260620 07:20:53.990972 32738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:53.990972 32737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:53.991374 32745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:53.993871 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:20:53.994438 32745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:54.006654 32745 catalog_manager.cc:1347] Generated new cluster ID: 71801a36b12844be96b86f7240ecbe45
I20260620 07:20:54.006726 32745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:54.009306 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:53.996839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:59459 (local address 127.0.0.1:33395)
0620 07:20:53.997038 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:53.997042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:53.997313 (+ 271us) server_negotiation.cc:407] Connection header received
0620 07:20:53.997498 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:53.997501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:53.997548 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:53.997597 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:53.998518 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:53.999409 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:54.000627 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:54.000994 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:54.001129 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:54.002880 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:54.002884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:54.002887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:54.002910 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:54.004606 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:54.005235 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:54.005239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:54.005241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:54.005341 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:54.005764 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:54.005768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:54.005770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:54.005972 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:54.006131 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:54.006480 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 07:20:54.006581 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:20:54.045941 32745 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:54.046921 32745 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:54.052142 32745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 445a2b561b874807af31f6d6dc32539e: Generated new TSK 0
I20260620 07:20:54.052800 32745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:54.060372 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 32663
W20260620 07:20:54.065586 32648 connection.cc:535] client connection to 127.0.0.1:33395 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33395): Connection reset by peer (error 104)
2026-06-20T07:20:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:54 dist-test-slave-j663 krb5kdc[32762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:54 dist-test-slave-j663 krb5kdc[32762](info): set up 2 sockets
Jun 20 07:20:54 dist-test-slave-j663 krb5kdc[32762](info): commencing operation
krb5kdc: starting...
W20260620 07:20:55.105317 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:55 dist-test-slave-j663 krb5kdc[32762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:20:55.220377 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:55.333691 307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:55.333956 307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:55.333998 307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:55.338160 307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:55.338215 307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:55.338234 307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:55.338251 307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:55.338266 307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:55.343611 307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:55.344573 307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:55.345633 307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:55.353675 312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:55.353667 313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:55.353808 307 server_base.cc:1034] running on GCE node
W20260620 07:20:55.353684 315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:55.354502 307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:55.355568 307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:55.356756 307 hybrid_clock.cc:648] HybridClock initialized: now 1781940055356728 us; error 40 us; skew 500 ppm
Jun 20 07:20:55 dist-test-slave-j663 krb5kdc[32762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:55.364576 307 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:20:55.365552 307 webserver.cc:460] Webserver started at http://127.0.0.1:35263/ using document root <none> and password file <none>
I20260620 07:20:55.366083 307 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:55.366143 307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:55.366307 307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:55.368319 307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "34bd1484fa334b65810e684b0f2f4866"
format_stamp: "Formatted at 2026-06-20 07:20:55 on dist-test-slave-j663"
server_key: "148b68ed25753c2353f5e7946b262870"
server_key_iv: "4026555d8412e043f186f6aad49b2d45"
server_key_version: "encryptionkey@0"
I20260620 07:20:55.368844 307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "34bd1484fa334b65810e684b0f2f4866"
format_stamp: "Formatted at 2026-06-20 07:20:55 on dist-test-slave-j663"
server_key: "148b68ed25753c2353f5e7946b262870"
server_key_iv: "4026555d8412e043f186f6aad49b2d45"
server_key_version: "encryptionkey@0"
I20260620 07:20:55.372929 307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:20:55.375594 322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:55.376849 307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:20:55.376948 307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "34bd1484fa334b65810e684b0f2f4866"
format_stamp: "Formatted at 2026-06-20 07:20:55 on dist-test-slave-j663"
server_key: "148b68ed25753c2353f5e7946b262870"
server_key_iv: "4026555d8412e043f186f6aad49b2d45"
server_key_version: "encryptionkey@0"
I20260620 07:20:55.377028 307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:55.399387 307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:55.402557 307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:55.402706 307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:55.409729 307 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38913
I20260620 07:20:55.409752 374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38913 every 8 connection(s)
I20260620 07:20:55.410646 307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:55.413316 375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:55.415982 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 307
I20260620 07:20:55.416126 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:55.416395 11758 external_mini_cluster.cc:1428] Setting key 3ea142c70f5f160979dfcdbe410c025a
I20260620 07:20:55.419050 375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866: Bootstrap starting.
I20260620 07:20:55.421201 375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:55.421902 375 log.cc:824] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:55.423645 375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866: No bootstrap required, opened a new log
I20260620 07:20:55.426167 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:55.417726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:43565 (local address 127.0.0.1:38913)
0620 07:20:55.418162 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:55.418176 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:55.418209 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:20:55.418819 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:55.418835 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:55.419181 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:55.419552 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:55.419980 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:55.420933 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:55.421899 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:55.422601 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:55.422783 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:55.423656 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:55.423662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:55.423684 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:55.424145 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:55.424235 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:55.424281 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:55.424434 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:55.424640 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 07:20:55.424772 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":136,"threads_started":1}
I20260620 07:20:55.427031 375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bd1484fa334b65810e684b0f2f4866" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38913 } }
I20260620 07:20:55.427263 375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:55.427313 375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34bd1484fa334b65810e684b0f2f4866, State: Initialized, Role: FOLLOWER
I20260620 07:20:55.427742 375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bd1484fa334b65810e684b0f2f4866" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38913 } }
I20260620 07:20:55.427879 375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:55.427932 375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:55.428012 375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:55.429014 375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bd1484fa334b65810e684b0f2f4866" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38913 } }
I20260620 07:20:55.429415 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:20:55.429412 375 leader_election.cc:302] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34bd1484fa334b65810e684b0f2f4866; no voters:
I20260620 07:20:55.429724 375 leader_election.cc:288] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:55.429847 380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:55.430158 380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [term 1 LEADER]: Becoming Leader. State: Replica: 34bd1484fa334b65810e684b0f2f4866, State: Running, Role: LEADER
I20260620 07:20:55.430550 380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bd1484fa334b65810e684b0f2f4866" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38913 } }
I20260620 07:20:55.431056 375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:55.431955 380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34bd1484fa334b65810e684b0f2f4866. Latest consensus state: current_term: 1 leader_uuid: "34bd1484fa334b65810e684b0f2f4866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bd1484fa334b65810e684b0f2f4866" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38913 } } }
I20260620 07:20:55.432138 380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:55.431957 381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34bd1484fa334b65810e684b0f2f4866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bd1484fa334b65810e684b0f2f4866" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38913 } } }
I20260620 07:20:55.432523 381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:55.433560 394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:55.436579 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:55.431599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:37161 (local address 127.0.0.1:38913)
0620 07:20:55.431692 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:55.431696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:55.432247 (+ 551us) server_negotiation.cc:407] Connection header received
0620 07:20:55.432436 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:55.432439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:55.432483 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:55.432534 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:55.432980 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:55.433860 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:55.434918 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:55.435238 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:55.435350 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:55.436119 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:55.436121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:55.436123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:55.436134 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:55.436155 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:55.436165 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:55.436232 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:55.436384 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 07:20:55.436409 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 07:20:55.437021 394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:55.441855 394 catalog_manager.cc:1347] Generated new cluster ID: c691d5b19f3244fd867c2bb7cb676f8d
I20260620 07:20:55.441918 394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:55.448586 394 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:55.449759 394 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:55.456125 394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34bd1484fa334b65810e684b0f2f4866: Generated new TSK 0
I20260620 07:20:55.456832 394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:55.464112 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 307
W20260620 07:20:55.469493 32760 connection.cc:535] client connection to 127.0.0.1:38913 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38913): Connection reset by peer (error 104)
2026-06-20T07:20:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:55 dist-test-slave-j663 krb5kdc[406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:55 dist-test-slave-j663 krb5kdc[406](info): set up 2 sockets
Jun 20 07:20:55 dist-test-slave-j663 krb5kdc[406](info): commencing operation
krb5kdc: starting...
W20260620 07:20:56.529335 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:56 dist-test-slave-j663 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:20:56.647768 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:56.755079 419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:56.755383 419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:56.755457 419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:56.759392 419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:56.759483 419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:56.759526 419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:56.759588 419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:56.759649 419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:56.764725 419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:56.765758 419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:56.766729 419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:20:56.775192 419 server_base.cc:1034] running on GCE node
W20260620 07:20:56.775095 425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:56.775085 424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:56.775146 427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:56.775949 419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:56.777034 419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:56.778261 419 hybrid_clock.cc:648] HybridClock initialized: now 1781940056778233 us; error 43 us; skew 500 ppm
Jun 20 07:20:56 dist-test-slave-j663 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:56.780958 419 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:20:56.781813 419 webserver.cc:460] Webserver started at http://127.0.0.1:36113/ using document root <none> and password file <none>
I20260620 07:20:56.782357 419 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:56.782446 419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:56.782612 419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:56.784643 419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "ff968b6d1ee34640bd7e810804fa7559"
format_stamp: "Formatted at 2026-06-20 07:20:56 on dist-test-slave-j663"
server_key: "ece957a912a522ab782e1ed2580f4b73"
server_key_iv: "0344970fe85df3bbc1a8a33cab53fadd"
server_key_version: "encryptionkey@0"
I20260620 07:20:56.785202 419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "ff968b6d1ee34640bd7e810804fa7559"
format_stamp: "Formatted at 2026-06-20 07:20:56 on dist-test-slave-j663"
server_key: "ece957a912a522ab782e1ed2580f4b73"
server_key_iv: "0344970fe85df3bbc1a8a33cab53fadd"
server_key_version: "encryptionkey@0"
I20260620 07:20:56.788607 419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:20:56.790952 434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:56.792237 419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 07:20:56.792387 419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "ff968b6d1ee34640bd7e810804fa7559"
format_stamp: "Formatted at 2026-06-20 07:20:56 on dist-test-slave-j663"
server_key: "ece957a912a522ab782e1ed2580f4b73"
server_key_iv: "0344970fe85df3bbc1a8a33cab53fadd"
server_key_version: "encryptionkey@0"
I20260620 07:20:56.792557 419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:56.797164 419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:56.800325 419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:56.800518 419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:56.808172 419 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36631
I20260620 07:20:56.808182 486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36631 every 8 connection(s)
I20260620 07:20:56.809049 419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:56.811666 487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:56.812943 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 419
I20260620 07:20:56.813129 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:56.813429 11758 external_mini_cluster.cc:1428] Setting key c6c37d83388f0881520434f872256159
I20260620 07:20:56.817473 487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559: Bootstrap starting.
I20260620 07:20:56.818271 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:56.814706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:45171 (local address 127.0.0.1:36631)
0620 07:20:56.815119 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:56.815134 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:56.815166 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:20:56.815769 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:56.815785 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:56.816046 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:56.816378 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:56.817141 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:56.817151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:56.817162 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:56.817173 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:56.817227 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:56.817245 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:56.817401 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:56.817471 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 07:20:56.817625 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260620 07:20:56.819804 487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:56.820662 487 log.cc:824] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:56.821069 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:20:56.822592 487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559: No bootstrap required, opened a new log
I20260620 07:20:56.826726 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:56.823107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:50477 (local address 127.0.0.1:36631)
0620 07:20:56.823209 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:56.823213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:56.825410 (+ 2197us) server_negotiation.cc:407] Connection header received
0620 07:20:56.825543 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:56.825545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:56.825575 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:56.825643 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:56.826341 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:56.826344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:56.826346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:56.826356 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:56.826379 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:56.826389 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:56.826452 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:56.826560 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 07:20:56.826592 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 07:20:56.826602 487 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff968b6d1ee34640bd7e810804fa7559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36631 } }
I20260620 07:20:56.826930 487 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:56.827004 487 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff968b6d1ee34640bd7e810804fa7559, State: Initialized, Role: FOLLOWER
I20260620 07:20:56.827433 487 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff968b6d1ee34640bd7e810804fa7559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36631 } }
I20260620 07:20:56.827622 487 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:56.827692 487 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:56.827798 487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:56.828655 487 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff968b6d1ee34640bd7e810804fa7559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36631 } }
I20260620 07:20:56.828989 487 leader_election.cc:302] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff968b6d1ee34640bd7e810804fa7559; no voters:
I20260620 07:20:56.829345 487 leader_election.cc:288] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:56.829411 497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:56.829617 497 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [term 1 LEADER]: Becoming Leader. State: Replica: ff968b6d1ee34640bd7e810804fa7559, State: Running, Role: LEADER
I20260620 07:20:56.829893 497 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff968b6d1ee34640bd7e810804fa7559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36631 } }
I20260620 07:20:56.830613 487 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:56.831554 498 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff968b6d1ee34640bd7e810804fa7559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff968b6d1ee34640bd7e810804fa7559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36631 } } }
I20260620 07:20:56.831694 498 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:56.832096 506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:56.832068 499 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff968b6d1ee34640bd7e810804fa7559. Latest consensus state: current_term: 1 leader_uuid: "ff968b6d1ee34640bd7e810804fa7559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff968b6d1ee34640bd7e810804fa7559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36631 } } }
I20260620 07:20:56.832276 499 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:56.835517 506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:56.841457 506 catalog_manager.cc:1347] Generated new cluster ID: c2f3ebfc40ec48169f9806755d25bce0
I20260620 07:20:56.841534 506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:56.876580 506 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:56.876955 506 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:56.886796 506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff968b6d1ee34640bd7e810804fa7559: Generated new TSK 0
I20260620 07:20:56.887441 506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:56.891832 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 419
2026-06-20T07:20:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:56 dist-test-slave-j663 krb5kdc[518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:56 dist-test-slave-j663 krb5kdc[518](info): set up 2 sockets
Jun 20 07:20:56 dist-test-slave-j663 krb5kdc[518](info): commencing operation
krb5kdc: starting...
W20260620 07:20:57.949501 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:58 dist-test-slave-j663 krb5kdc[518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:20:58.065948 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:58.177435 531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:58.177793 531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:58.177896 531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:58.182036 531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:58.182109 531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:58.182157 531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:58.182191 531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:58.182221 531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:58.187274 531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:58.188225 531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:58.189235 531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:58.197392 537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:58.197403 536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:58.197549 531 server_base.cc:1034] running on GCE node
W20260620 07:20:58.197432 539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:58.198132 531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:58.199146 531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:58.200358 531 hybrid_clock.cc:648] HybridClock initialized: now 1781940058200339 us; error 28 us; skew 500 ppm
Jun 20 07:20:58 dist-test-slave-j663 krb5kdc[518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:58.203209 531 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:20:58.204028 531 webserver.cc:460] Webserver started at http://127.0.0.1:41311/ using document root <none> and password file <none>
I20260620 07:20:58.204564 531 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:58.204653 531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:58.204900 531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:58.206900 531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "dafb3e5c75fb40ba98e17ccc8ed62452"
format_stamp: "Formatted at 2026-06-20 07:20:58 on dist-test-slave-j663"
server_key: "9505d59862f40d08cb93e06d5e20391b"
server_key_iv: "03a3a9026d3c3d8094e49d14bc029780"
server_key_version: "encryptionkey@0"
I20260620 07:20:58.207441 531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "dafb3e5c75fb40ba98e17ccc8ed62452"
format_stamp: "Formatted at 2026-06-20 07:20:58 on dist-test-slave-j663"
server_key: "9505d59862f40d08cb93e06d5e20391b"
server_key_iv: "03a3a9026d3c3d8094e49d14bc029780"
server_key_version: "encryptionkey@0"
I20260620 07:20:58.211120 531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:20:58.213511 546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:58.214711 531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:20:58.214866 531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "dafb3e5c75fb40ba98e17ccc8ed62452"
format_stamp: "Formatted at 2026-06-20 07:20:58 on dist-test-slave-j663"
server_key: "9505d59862f40d08cb93e06d5e20391b"
server_key_iv: "03a3a9026d3c3d8094e49d14bc029780"
server_key_version: "encryptionkey@0"
I20260620 07:20:58.214973 531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:58.227586 531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:58.230631 531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:58.230834 531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:58.237277 531 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34595
I20260620 07:20:58.237303 598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34595 every 8 connection(s)
I20260620 07:20:58.238147 531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:58.240772 599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:58.240834 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 531
I20260620 07:20:58.240964 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:58.241304 11758 external_mini_cluster.cc:1428] Setting key bf2fffb248de2722e1b9ca47740a1331
I20260620 07:20:58.248385 599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452: Bootstrap starting.
I20260620 07:20:58.250895 599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452: Neither blocks nor log segments found. Creating new log.
Jun 20 07:20:58 dist-test-slave-j663 krb5kdc[518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:20:58.251652 599 log.cc:824] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:58.253500 599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452: No bootstrap required, opened a new log
I20260620 07:20:58.256633 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:58.242619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:59909 (local address 127.0.0.1:34595)
0620 07:20:58.243120 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:58.243140 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:58.243181 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:20:58.244613 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:58.244637 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:58.245066 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:58.245466 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:58.246412 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:58.247874 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:58.248754 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:58.249453 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:58.249633 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:58.252131 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:58.252137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:58.252149 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:58.252176 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:58.253895 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:58.254395 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:58.254399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:58.254403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:58.254506 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:58.254849 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:58.254852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:58.254854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:58.255227 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:58.255443 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:58.255757 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 07:20:58.255963 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":151,"threads_started":1}
I20260620 07:20:58.256997 599 raft_consensus.cc:348] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dafb3e5c75fb40ba98e17ccc8ed62452" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34595 } }
I20260620 07:20:58.257292 599 raft_consensus.cc:374] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:58.257385 599 raft_consensus.cc:729] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dafb3e5c75fb40ba98e17ccc8ed62452, State: Initialized, Role: FOLLOWER
I20260620 07:20:58.257853 599 consensus_queue.cc:260] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dafb3e5c75fb40ba98e17ccc8ed62452" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34595 } }
I20260620 07:20:58.258018 599 raft_consensus.cc:388] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:58.258082 599 raft_consensus.cc:482] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:58.258173 599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:58.259250 599 raft_consensus.cc:504] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dafb3e5c75fb40ba98e17ccc8ed62452" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34595 } }
I20260620 07:20:58.259618 599 leader_election.cc:302] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dafb3e5c75fb40ba98e17ccc8ed62452; no voters:
I20260620 07:20:58.259785 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:20:58.259989 599 leader_election.cc:288] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:58.260090 604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:58.260285 604 raft_consensus.cc:686] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [term 1 LEADER]: Becoming Leader. State: Replica: dafb3e5c75fb40ba98e17ccc8ed62452, State: Running, Role: LEADER
I20260620 07:20:58.260653 604 consensus_queue.cc:237] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dafb3e5c75fb40ba98e17ccc8ed62452" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34595 } }
I20260620 07:20:58.261262 599 sys_catalog.cc:564] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:58.262253 605 sys_catalog.cc:455] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dafb3e5c75fb40ba98e17ccc8ed62452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dafb3e5c75fb40ba98e17ccc8ed62452" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34595 } } }
I20260620 07:20:58.262326 606 sys_catalog.cc:455] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dafb3e5c75fb40ba98e17ccc8ed62452. Latest consensus state: current_term: 1 leader_uuid: "dafb3e5c75fb40ba98e17ccc8ed62452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dafb3e5c75fb40ba98e17ccc8ed62452" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34595 } } }
I20260620 07:20:58.262377 605 sys_catalog.cc:458] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:58.262394 606 sys_catalog.cc:458] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:58.262754 618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:58.265923 618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:58.271687 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:58.262747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:58989 (local address 127.0.0.1:34595)
0620 07:20:58.262915 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:58.262919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:58.263157 (+ 238us) server_negotiation.cc:407] Connection header received
0620 07:20:58.263323 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:58.263327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:58.263373 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:58.263424 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:58.264290 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:58.265125 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:58.266250 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:58.266552 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:58.266624 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:58.268304 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:58.268308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:58.268310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:20:58.268333 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:58.269844 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:58.270407 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:58.270411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:58.270413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:58.270496 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:20:58.270849 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:20:58.270852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:20:58.270854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:20:58.271016 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:58.271163 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:58.271429 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 07:20:58.271521 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 07:20:58.271811 618 catalog_manager.cc:1347] Generated new cluster ID: 606d6e05d11d4eb4831aec03850b76e8
I20260620 07:20:58.271859 618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:58.299418 618 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:58.300347 618 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:58.305788 618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dafb3e5c75fb40ba98e17ccc8ed62452: Generated new TSK 0
I20260620 07:20:58.306600 618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:58.320428 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 531
W20260620 07:20:58.326262 516 connection.cc:535] client connection to 127.0.0.1:34595 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34595): Connection reset by peer (error 104)
2026-06-20T07:20:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:58 dist-test-slave-j663 krb5kdc[630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:58 dist-test-slave-j663 krb5kdc[630](info): set up 2 sockets
Jun 20 07:20:58 dist-test-slave-j663 krb5kdc[630](info): commencing operation
krb5kdc: starting...
W20260620 07:20:59.377379 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:20:59 dist-test-slave-j663 krb5kdc[630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:20:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:20:59.496536 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:20:59.607328 643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:20:59.607594 643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:20:59.607636 643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:20:59.611608 643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:20:59.611663 643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:20:59.611719 643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:20:59.611759 643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:20:59.611797 643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:20:59.616878 643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42487
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:20:59.617906 643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:20:59.618897 643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:20:59.627620 649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:59.627674 643 server_base.cc:1034] running on GCE node
W20260620 07:20:59.627626 648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:20:59.627638 651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:20:59.628413 643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:20:59.629487 643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:20:59.630662 643 hybrid_clock.cc:648] HybridClock initialized: now 1781940059630628 us; error 46 us; skew 500 ppm
Jun 20 07:20:59 dist-test-slave-j663 krb5kdc[630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:20:59.633386 643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:20:59.634294 643 webserver.cc:460] Webserver started at http://127.0.0.1:40449/ using document root <none> and password file <none>
I20260620 07:20:59.634852 643 fs_manager.cc:359] Metadata directory not provided
I20260620 07:20:59.634944 643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:20:59.635178 643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:20:59.637281 643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "feb0bf4e64724d1ba3486297ebfdf535"
format_stamp: "Formatted at 2026-06-20 07:20:59 on dist-test-slave-j663"
server_key: "8e15c9bd472b47000bbe7610ffb7042e"
server_key_iv: "7a57994866a9cb96df938dfbed5bc764"
server_key_version: "encryptionkey@0"
I20260620 07:20:59.637840 643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "feb0bf4e64724d1ba3486297ebfdf535"
format_stamp: "Formatted at 2026-06-20 07:20:59 on dist-test-slave-j663"
server_key: "8e15c9bd472b47000bbe7610ffb7042e"
server_key_iv: "7a57994866a9cb96df938dfbed5bc764"
server_key_version: "encryptionkey@0"
I20260620 07:20:59.641448 643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:20:59.643774 658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:20:59.645061 643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:20:59.645267 643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "feb0bf4e64724d1ba3486297ebfdf535"
format_stamp: "Formatted at 2026-06-20 07:20:59 on dist-test-slave-j663"
server_key: "8e15c9bd472b47000bbe7610ffb7042e"
server_key_iv: "7a57994866a9cb96df938dfbed5bc764"
server_key_version: "encryptionkey@0"
I20260620 07:20:59.645380 643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:20:59.674175 643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:20:59.677435 643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:20:59.677647 643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:20:59.684608 643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42487
I20260620 07:20:59.684633 710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42487 every 8 connection(s)
I20260620 07:20:59.685597 643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:20:59.688612 711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:20:59.692341 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 643
I20260620 07:20:59.692489 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:20:59.692773 11758 external_mini_cluster.cc:1428] Setting key a43fe3976d016d2a21945c3ad59d2e04
I20260620 07:20:59.694480 711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535: Bootstrap starting.
I20260620 07:20:59.696596 711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535: Neither blocks nor log segments found. Creating new log.
I20260620 07:20:59.697378 711 log.cc:824] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535: Log is configured to *not* fsync() on all Append() calls
I20260620 07:20:59.699307 711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535: No bootstrap required, opened a new log
I20260620 07:20:59.701828 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:59.694175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:59713 (local address 127.0.0.1:42487)
0620 07:20:59.694655 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:59.694675 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:59.694713 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:20:59.695328 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:59.695344 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:59.695666 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:59.695993 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:59.696401 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:59.697517 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:59.698577 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:59.699133 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:59.699311 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:59.700194 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:59.700200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:59.700212 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:59.700589 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:59.700642 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:59.700665 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:59.700803 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:59.700997 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 07:20:59.701170 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":145,"threads_started":1}
I20260620 07:20:59.703019 711 raft_consensus.cc:348] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb0bf4e64724d1ba3486297ebfdf535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } }
I20260620 07:20:59.703711 711 raft_consensus.cc:374] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:20:59.703791 711 raft_consensus.cc:729] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb0bf4e64724d1ba3486297ebfdf535, State: Initialized, Role: FOLLOWER
I20260620 07:20:59.704313 711 consensus_queue.cc:260] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb0bf4e64724d1ba3486297ebfdf535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } }
I20260620 07:20:59.704485 711 raft_consensus.cc:388] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:20:59.704553 711 raft_consensus.cc:482] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:20:59.704653 711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:20:59.704892 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:20:59.705729 711 raft_consensus.cc:504] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb0bf4e64724d1ba3486297ebfdf535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } }
I20260620 07:20:59.706008 711 leader_election.cc:302] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feb0bf4e64724d1ba3486297ebfdf535; no voters:
I20260620 07:20:59.706317 711 leader_election.cc:288] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:20:59.706374 719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:20:59.706565 719 raft_consensus.cc:686] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [term 1 LEADER]: Becoming Leader. State: Replica: feb0bf4e64724d1ba3486297ebfdf535, State: Running, Role: LEADER
I20260620 07:20:59.706887 719 consensus_queue.cc:237] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb0bf4e64724d1ba3486297ebfdf535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } }
I20260620 07:20:59.707547 711 sys_catalog.cc:564] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:20:59.708034 722 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feb0bf4e64724d1ba3486297ebfdf535. Latest consensus state: current_term: 1 leader_uuid: "feb0bf4e64724d1ba3486297ebfdf535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb0bf4e64724d1ba3486297ebfdf535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } } }
I20260620 07:20:59.708173 722 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:59.708501 721 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feb0bf4e64724d1ba3486297ebfdf535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb0bf4e64724d1ba3486297ebfdf535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } } }
I20260620 07:20:59.708674 721 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535 [sys.catalog]: This master's current role is: LEADER
I20260620 07:20:59.708935 726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:20:59.711964 726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:20:59.712707 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:20:59.707335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:45227 (local address 127.0.0.1:42487)
0620 07:20:59.707428 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 07:20:59.707432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:20:59.707445 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:20:59.707477 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:20:59.707479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:20:59.707519 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:20:59.707602 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:20:59.708424 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:59.709261 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:59.710542 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:20:59.710871 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:20:59.710971 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:20:59.711911 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:20:59.711914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:20:59.711917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:20:59.711930 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:20:59.711961 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:20:59.711972 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:20:59.712069 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:20:59.712525 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 07:20:59.712563 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 07:20:59.717446 726 catalog_manager.cc:1347] Generated new cluster ID: ff1dbe51f1ba4c51b7a57d4fedf4dd3d
I20260620 07:20:59.717507 726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:20:59.728935 726 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:20:59.729869 726 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:20:59.740432 726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P feb0bf4e64724d1ba3486297ebfdf535: Generated new TSK 0
I20260620 07:20:59.741366 726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:20:59.748612 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 643
W20260620 07:20:59.753679 628 connection.cc:535] client connection to 127.0.0.1:42487 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42487): Connection reset by peer (error 104)
2026-06-20T07:20:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:20:59 dist-test-slave-j663 krb5kdc[742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:20:59 dist-test-slave-j663 krb5kdc[742](info): set up 2 sockets
Jun 20 07:20:59 dist-test-slave-j663 krb5kdc[742](info): commencing operation
krb5kdc: starting...
W20260620 07:21:00.805527 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:21:00 dist-test-slave-j663 krb5kdc[742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:21:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:21:00.932596 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:21:01.041800 755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:01.042094 755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:01.042167 755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:21:01.046131 755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:01.046213 755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:01.046257 755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:01.046291 755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:01.046324 755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:01.051412 755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:01.052382 755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:01.053395 755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:01.061398 763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:01.061376 761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:01.061420 760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:01.061690 755 server_base.cc:1034] running on GCE node
I20260620 07:21:01.062235 755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:01.063249 755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:01.064445 755 hybrid_clock.cc:648] HybridClock initialized: now 1781940061064423 us; error 36 us; skew 500 ppm
Jun 20 07:21:01 dist-test-slave-j663 krb5kdc[742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:01.071219 755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:21:01.072052 755 webserver.cc:460] Webserver started at http://127.0.0.1:35373/ using document root <none> and password file <none>
I20260620 07:21:01.072597 755 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:01.072686 755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:01.072918 755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:01.074930 755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "b417d856a5f743279f5739cece8cf9f9"
format_stamp: "Formatted at 2026-06-20 07:21:01 on dist-test-slave-j663"
server_key: "bda6113a8d4411a99f95a236b096d344"
server_key_iv: "08f33314cbc9dc3f119ea394de90c1c9"
server_key_version: "encryptionkey@0"
I20260620 07:21:01.075484 755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "b417d856a5f743279f5739cece8cf9f9"
format_stamp: "Formatted at 2026-06-20 07:21:01 on dist-test-slave-j663"
server_key: "bda6113a8d4411a99f95a236b096d344"
server_key_iv: "08f33314cbc9dc3f119ea394de90c1c9"
server_key_version: "encryptionkey@0"
I20260620 07:21:01.079128 755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 07:21:01.081722 770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:01.082892 755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:21:01.083039 755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "b417d856a5f743279f5739cece8cf9f9"
format_stamp: "Formatted at 2026-06-20 07:21:01 on dist-test-slave-j663"
server_key: "bda6113a8d4411a99f95a236b096d344"
server_key_iv: "08f33314cbc9dc3f119ea394de90c1c9"
server_key_version: "encryptionkey@0"
I20260620 07:21:01.083148 755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:01.087455 755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:01.090199 755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:01.090402 755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:01.097019 755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40843
I20260620 07:21:01.097110 822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40843 every 8 connection(s)
I20260620 07:21:01.097985 755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:21:01.100646 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:01.106258 823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9: Bootstrap starting.
I20260620 07:21:01.108117 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 755
I20260620 07:21:01.108275 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:21:01.108556 11758 external_mini_cluster.cc:1428] Setting key 978c3b10a76e3b83b5bf881c9abcf96e
I20260620 07:21:01.108767 823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9: Neither blocks nor log segments found. Creating new log.
I20260620 07:21:01.109640 823 log.cc:824] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:21:01.111831 823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9: No bootstrap required, opened a new log
I20260620 07:21:01.114502 826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:01.109647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:56185 (local address 127.0.0.1:40843)
0620 07:21:01.110129 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:01.110153 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:01.110194 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:21:01.110999 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:01.111021 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:01.111364 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:01.111759 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:01.112564 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:01.112569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:01.112584 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:21:01.112596 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:01.112659 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:21:01.112686 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:01.112831 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:01.113442 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 07:21:01.113599 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":132,"threads_started":1}
I20260620 07:21:01.116698 823 raft_consensus.cc:348] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b417d856a5f743279f5739cece8cf9f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40843 } }
I20260620 07:21:01.116962 823 raft_consensus.cc:374] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:21:01.117048 823 raft_consensus.cc:729] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b417d856a5f743279f5739cece8cf9f9, State: Initialized, Role: FOLLOWER
I20260620 07:21:01.117409 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:21:01.117532 823 consensus_queue.cc:260] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b417d856a5f743279f5739cece8cf9f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40843 } }
I20260620 07:21:01.117690 823 raft_consensus.cc:388] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:21:01.117751 823 raft_consensus.cc:482] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:21:01.117841 823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:01.119045 823 raft_consensus.cc:504] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b417d856a5f743279f5739cece8cf9f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40843 } }
I20260620 07:21:01.119405 823 leader_election.cc:302] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b417d856a5f743279f5739cece8cf9f9; no voters:
I20260620 07:21:01.119787 823 leader_election.cc:288] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:21:01.119942 832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:21:01.120146 832 raft_consensus.cc:686] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [term 1 LEADER]: Becoming Leader. State: Replica: b417d856a5f743279f5739cece8cf9f9, State: Running, Role: LEADER
I20260620 07:21:01.120447 832 consensus_queue.cc:237] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b417d856a5f743279f5739cece8cf9f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40843 } }
I20260620 07:21:01.121404 823 sys_catalog.cc:564] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:21:01.121585 826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:01.119700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:57547 (local address 127.0.0.1:40843)
0620 07:21:01.119832 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:01.119838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:01.120066 (+ 228us) server_negotiation.cc:407] Connection header received
0620 07:21:01.120268 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:01.120272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:01.120307 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:01.120362 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:01.121154 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:01.121158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:01.121160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:21:01.121172 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:01.121198 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:21:01.121217 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:01.121308 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:01.121383 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 07:21:01.121424 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:21:01.121652 832 sys_catalog.cc:455] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b417d856a5f743279f5739cece8cf9f9. Latest consensus state: current_term: 1 leader_uuid: "b417d856a5f743279f5739cece8cf9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b417d856a5f743279f5739cece8cf9f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40843 } } }
I20260620 07:21:01.121763 832 sys_catalog.cc:458] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:01.122336 835 sys_catalog.cc:455] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b417d856a5f743279f5739cece8cf9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b417d856a5f743279f5739cece8cf9f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40843 } } }
I20260620 07:21:01.122496 835 sys_catalog.cc:458] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:01.122709 837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:21:01.125898 837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:21:01.131858 837 catalog_manager.cc:1347] Generated new cluster ID: 2d33e31ab3ae44dc82daa2f7426f24da
I20260620 07:21:01.131948 837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:21:01.139807 837 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:21:01.140345 837 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:21:01.148756 837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b417d856a5f743279f5739cece8cf9f9: Generated new TSK 0
I20260620 07:21:01.149410 837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:21:01.159201 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 755
2026-06-20T07:21:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:21:01 dist-test-slave-j663 krb5kdc[854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:21:01 dist-test-slave-j663 krb5kdc[854](info): set up 2 sockets
Jun 20 07:21:01 dist-test-slave-j663 krb5kdc[854](info): commencing operation
krb5kdc: starting...
W20260620 07:21:02.233181 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:21:02 dist-test-slave-j663 krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:21:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:21:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:21:02.357620 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:44653
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:44653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:21:02.469211 867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:02.469448 867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:02.469488 867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:21:02.473466 867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:02.473515 867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:02.473534 867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:02.473572 867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:02.473606 867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:02.478754 867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:44653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:44653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:02.479771 867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:02.480887 867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:02.489220 875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:02.489171 872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:02.489358 867 server_base.cc:1034] running on GCE node
W20260620 07:21:02.489395 873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:02.490043 867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:02.491114 867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:02.492300 867 hybrid_clock.cc:648] HybridClock initialized: now 1781940062492282 us; error 33 us; skew 500 ppm
Jun 20 07:21:02 dist-test-slave-j663 krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940062, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:02.495325 867 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:21:02.496287 867 webserver.cc:460] Webserver started at http://127.11.123.190:41509/ using document root <none> and password file <none>
I20260620 07:21:02.496850 867 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:02.496944 867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:02.497229 867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:02.499253 867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "ec8ef1a3688042949940c84a9824f78f"
format_stamp: "Formatted at 2026-06-20 07:21:02 on dist-test-slave-j663"
server_key: "ff6b3872ec1811f4ab0350ea011c1dd1"
server_key_iv: "718d563776d08923e142d347f1cdff53"
server_key_version: "encryptionkey@0"
I20260620 07:21:02.499843 867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "ec8ef1a3688042949940c84a9824f78f"
format_stamp: "Formatted at 2026-06-20 07:21:02 on dist-test-slave-j663"
server_key: "ff6b3872ec1811f4ab0350ea011c1dd1"
server_key_iv: "718d563776d08923e142d347f1cdff53"
server_key_version: "encryptionkey@0"
I20260620 07:21:02.503543 867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:21:02.505980 882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:02.507282 867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:21:02.507429 867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "ec8ef1a3688042949940c84a9824f78f"
format_stamp: "Formatted at 2026-06-20 07:21:02 on dist-test-slave-j663"
server_key: "ff6b3872ec1811f4ab0350ea011c1dd1"
server_key_iv: "718d563776d08923e142d347f1cdff53"
server_key_version: "encryptionkey@0"
I20260620 07:21:02.507542 867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:02.540311 867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:02.543270 867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:02.543506 867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:02.550853 867 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:44653
I20260620 07:21:02.550879 934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:44653 every 8 connection(s)
I20260620 07:21:02.551774 867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:21:02.553838 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 867
I20260620 07:21:02.553941 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:21:02.554234 11758 external_mini_cluster.cc:1428] Setting key d5411258c6323bde81297ac02b3637fb
I20260620 07:21:02.554769 935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:02.561863 935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f: Bootstrap starting.
Jun 20 07:21:02 dist-test-slave-j663 krb5kdc[854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:21:02.564499 935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f: Neither blocks nor log segments found. Creating new log.
I20260620 07:21:02.565588 935 log.cc:824] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:21:02.568015 935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f: No bootstrap required, opened a new log
I20260620 07:21:02.569422 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:02.555610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60959 (local address 127.11.123.190:44653)
0620 07:21:02.556037 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:02.556060 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:02.556095 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:21:02.556726 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:02.556742 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:02.557132 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:02.557426 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:02.558376 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:02.559326 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:02.560498 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:02.560938 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:02.561053 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:02.563838 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:02.563844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:02.563860 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:02.563891 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:02.565958 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:02.566622 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:02.566627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:02.566633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:02.566819 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:02.567135 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:02.567138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:02.567139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:02.567633 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:02.567850 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:02.568069 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 07:21:02.568489 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":174,"threads_started":1}
I20260620 07:21:02.572530 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:21:02.572432 935 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8ef1a3688042949940c84a9824f78f" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44653 } }
I20260620 07:21:02.572726 935 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:21:02.572805 935 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8ef1a3688042949940c84a9824f78f, State: Initialized, Role: FOLLOWER
I20260620 07:21:02.573292 935 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8ef1a3688042949940c84a9824f78f" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44653 } }
I20260620 07:21:02.573444 935 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:21:02.573537 935 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:21:02.573634 935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:02.574556 935 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8ef1a3688042949940c84a9824f78f" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44653 } }
I20260620 07:21:02.574927 935 leader_election.cc:302] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec8ef1a3688042949940c84a9824f78f; no voters:
I20260620 07:21:02.575309 935 leader_election.cc:288] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:21:02.575450 945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:21:02.575696 945 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [term 1 LEADER]: Becoming Leader. State: Replica: ec8ef1a3688042949940c84a9824f78f, State: Running, Role: LEADER
I20260620 07:21:02.576056 945 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8ef1a3688042949940c84a9824f78f" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44653 } }
I20260620 07:21:02.576691 935 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:21:02.577674 947 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec8ef1a3688042949940c84a9824f78f. Latest consensus state: current_term: 1 leader_uuid: "ec8ef1a3688042949940c84a9824f78f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8ef1a3688042949940c84a9824f78f" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44653 } } }
I20260620 07:21:02.577814 947 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:02.579193 946 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec8ef1a3688042949940c84a9824f78f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8ef1a3688042949940c84a9824f78f" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44653 } } }
I20260620 07:21:02.579533 946 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f [sys.catalog]: This master's current role is: LEADER
W20260620 07:21:02.582155 961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:21:02.582237 961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:21:02.582307 954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:21:02.583181 954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:21:02.584060 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:02.574740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50667 (local address 127.11.123.190:44653)
0620 07:21:02.574852 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:02.574856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:02.575014 (+ 158us) server_negotiation.cc:407] Connection header received
0620 07:21:02.575123 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:02.575125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:02.575174 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:02.575267 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:02.576096 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:02.576871 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:02.578066 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:02.578420 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:02.578522 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:02.580376 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:02.580379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:02.580381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:02.580402 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:02.582036 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:02.582603 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:02.582607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:02.582609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:02.582698 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:02.583079 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:02.583082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:02.583084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:02.583355 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:02.583504 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:02.583812 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 07:21:02.583909 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 07:21:02.588330 954 catalog_manager.cc:1347] Generated new cluster ID: 383a271924904ce3996ad59d74675e25
I20260620 07:21:02.588393 954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:21:02.613381 954 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:21:02.614627 954 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:21:02.625669 954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec8ef1a3688042949940c84a9824f78f: Generated new TSK 0
I20260620 07:21:02.626456 954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:21:02.640765 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 867
W20260620 07:21:02.646121 852 connection.cc:535] client connection to 127.11.123.190:44653 recv error: Network error: failed to read from TLS socket (remote: 127.11.123.190:44653): Connection reset by peer (error 104)
2026-06-20T07:21:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:21:02 dist-test-slave-j663 krb5kdc[966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:21:02 dist-test-slave-j663 krb5kdc[966](info): set up 2 sockets
Jun 20 07:21:02 dist-test-slave-j663 krb5kdc[966](info): commencing operation
krb5kdc: starting...
W20260620 07:21:03.689370 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:21:03 dist-test-slave-j663 krb5kdc[966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:21:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:21:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:21:03.814081 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:42225
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:42225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:21:03.925851 979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:03.926121 979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:03.926162 979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:21:03.930337 979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:03.930392 979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:03.930411 979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:03.930428 979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:03.930442 979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:03.935652 979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:42225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:42225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:03.936604 979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:03.937613 979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:03.945881 987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:03.945909 985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:03.946004 979 server_base.cc:1034] running on GCE node
W20260620 07:21:03.945905 984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:03.946630 979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:03.947723 979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:03.948889 979 hybrid_clock.cc:648] HybridClock initialized: now 1781940063948869 us; error 31 us; skew 500 ppm
Jun 20 07:21:03 dist-test-slave-j663 krb5kdc[966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940063, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:03.951673 979 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:21:03.952493 979 webserver.cc:460] Webserver started at http://127.11.123.190:37013/ using document root <none> and password file <none>
I20260620 07:21:03.953035 979 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:03.953156 979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:03.953377 979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:03.955428 979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "1933a4dbdb704c779fad38bd7afe9414"
format_stamp: "Formatted at 2026-06-20 07:21:03 on dist-test-slave-j663"
server_key: "7f4a88d7379b1b8ff16a2ca65735658f"
server_key_iv: "f6bd9bd936cbf87c88593451a84d3787"
server_key_version: "encryptionkey@0"
I20260620 07:21:03.956020 979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "1933a4dbdb704c779fad38bd7afe9414"
format_stamp: "Formatted at 2026-06-20 07:21:03 on dist-test-slave-j663"
server_key: "7f4a88d7379b1b8ff16a2ca65735658f"
server_key_iv: "f6bd9bd936cbf87c88593451a84d3787"
server_key_version: "encryptionkey@0"
I20260620 07:21:03.959756 979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260620 07:21:03.962208 994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:03.963428 979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:21:03.963569 979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "1933a4dbdb704c779fad38bd7afe9414"
format_stamp: "Formatted at 2026-06-20 07:21:03 on dist-test-slave-j663"
server_key: "7f4a88d7379b1b8ff16a2ca65735658f"
server_key_iv: "f6bd9bd936cbf87c88593451a84d3787"
server_key_version: "encryptionkey@0"
I20260620 07:21:03.963678 979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:03.975015 979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:03.977896 979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:03.978094 979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:03.984752 979 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:42225
I20260620 07:21:03.984813 1046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:42225 every 8 connection(s)
I20260620 07:21:03.985785 979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:21:03.988445 1047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:03.991048 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 979
I20260620 07:21:03.991201 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:21:03.991493 11758 external_mini_cluster.cc:1428] Setting key 5560a2fd1db131a5db40068c7d1f4fa5
I20260620 07:21:03.994966 1047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414: Bootstrap starting.
I20260620 07:21:03.997506 1047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414: Neither blocks nor log segments found. Creating new log.
I20260620 07:21:03.998215 1047 log.cc:824] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414: Log is configured to *not* fsync() on all Append() calls
I20260620 07:21:04.000332 1047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414: No bootstrap required, opened a new log
I20260620 07:21:04.000999 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:03.992756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49525 (local address 127.11.123.190:42225)
0620 07:21:03.993221 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:03.993239 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:03.993280 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:21:03.994103 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:03.994126 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:03.994531 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:03.994895 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:03.995287 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:03.996701 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:03.997690 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:03.998161 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:03.998355 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:03.999256 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:03.999261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:03.999273 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:21:03.999674 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:03.999726 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:21:03.999749 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:03.999851 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:04.000087 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 07:21:04.000223 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":109,"threads_started":1}
I20260620 07:21:04.004117 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:21:04.004415 1047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1933a4dbdb704c779fad38bd7afe9414" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 42225 } }
I20260620 07:21:04.004621 1047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:21:04.004693 1047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1933a4dbdb704c779fad38bd7afe9414, State: Initialized, Role: FOLLOWER
I20260620 07:21:04.005160 1047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1933a4dbdb704c779fad38bd7afe9414" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 42225 } }
I20260620 07:21:04.005283 1047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:21:04.005365 1047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:21:04.005465 1047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:04.006494 1047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1933a4dbdb704c779fad38bd7afe9414" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 42225 } }
I20260620 07:21:04.006901 1047 leader_election.cc:302] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1933a4dbdb704c779fad38bd7afe9414; no voters:
I20260620 07:21:04.007264 1047 leader_election.cc:288] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:21:04.007369 1057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:21:04.007584 1057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [term 1 LEADER]: Becoming Leader. State: Replica: 1933a4dbdb704c779fad38bd7afe9414, State: Running, Role: LEADER
I20260620 07:21:04.007918 1057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1933a4dbdb704c779fad38bd7afe9414" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 42225 } }
I20260620 07:21:04.008514 1047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:21:04.009008 1057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1933a4dbdb704c779fad38bd7afe9414. Latest consensus state: current_term: 1 leader_uuid: "1933a4dbdb704c779fad38bd7afe9414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1933a4dbdb704c779fad38bd7afe9414" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 42225 } } }
I20260620 07:21:04.009224 1057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:04.009469 1058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1933a4dbdb704c779fad38bd7afe9414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1933a4dbdb704c779fad38bd7afe9414" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 42225 } } }
I20260620 07:21:04.009613 1058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:04.009680 1064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:21:04.011113 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:04.006256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58959 (local address 127.11.123.190:42225)
0620 07:21:04.006392 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:04.006399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:04.006508 (+ 109us) server_negotiation.cc:407] Connection header received
0620 07:21:04.006580 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:04.006585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:04.006626 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:04.006706 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:04.006926 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:04.007701 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:04.008804 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:04.009156 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:04.009287 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:04.010384 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:04.010391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:04.010397 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:21:04.010413 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:04.010443 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:21:04.010463 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:04.010553 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:04.010905 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 07:21:04.010944 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:21:04.013161 1064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:21:04.018882 1064 catalog_manager.cc:1347] Generated new cluster ID: 74b69f73029748979cdaa6d32d9362b2
I20260620 07:21:04.018985 1064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:21:04.031687 1064 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:21:04.032642 1064 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:21:04.043537 1064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1933a4dbdb704c779fad38bd7afe9414: Generated new TSK 0
I20260620 07:21:04.044366 1064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:21:04.056600 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 979
W20260620 07:21:04.062171 964 connection.cc:535] client connection to 127.11.123.190:42225 recv error: Network error: failed to read from TLS socket (remote: 127.11.123.190:42225): Connection reset by peer (error 104)
2026-06-20T07:21:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:21:04 dist-test-slave-j663 krb5kdc[1078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:21:04 dist-test-slave-j663 krb5kdc[1078](info): set up 2 sockets
Jun 20 07:21:04 dist-test-slave-j663 krb5kdc[1078](info): commencing operation
krb5kdc: starting...
W20260620 07:21:05.113364 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:21:05 dist-test-slave-j663 krb5kdc[1078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:21:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:21:05.232402 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:35417
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:40563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:35417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:21:05.346613 1091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:05.347007 1091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:05.347080 1091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:21:05.352010 1091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:05.352082 1091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:05.352101 1091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:05.352118 1091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:05.352133 1091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:05.357398 1091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:40563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:35417
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:35417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:05.358355 1091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:05.359337 1091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:05.367686 1097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:05.367710 1099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:05.367666 1096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:05.368034 1091 server_base.cc:1034] running on GCE node
I20260620 07:21:05.368631 1091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:05.369742 1091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:05.370955 1091 hybrid_clock.cc:648] HybridClock initialized: now 1781940065370918 us; error 48 us; skew 500 ppm
Jun 20 07:21:05 dist-test-slave-j663 krb5kdc[1078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940065, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:05.374439 1091 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:21:05.375380 1091 webserver.cc:460] Webserver started at http://127.11.123.190:35181/ using document root <none> and password file <none>
I20260620 07:21:05.375914 1091 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:05.376003 1091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:05.376235 1091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:05.378270 1091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "52aae42f9925435c9e81d6649b336488"
format_stamp: "Formatted at 2026-06-20 07:21:05 on dist-test-slave-j663"
server_key: "bca4618c700b1566ffb68c69e4682307"
server_key_iv: "5b8aa432580f9e521e39d3f69b1b8230"
server_key_version: "encryptionkey@0"
I20260620 07:21:05.378816 1091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "52aae42f9925435c9e81d6649b336488"
format_stamp: "Formatted at 2026-06-20 07:21:05 on dist-test-slave-j663"
server_key: "bca4618c700b1566ffb68c69e4682307"
server_key_iv: "5b8aa432580f9e521e39d3f69b1b8230"
server_key_version: "encryptionkey@0"
I20260620 07:21:05.382351 1091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:21:05.384583 1106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:05.385845 1091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:21:05.385982 1091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "52aae42f9925435c9e81d6649b336488"
format_stamp: "Formatted at 2026-06-20 07:21:05 on dist-test-slave-j663"
server_key: "bca4618c700b1566ffb68c69e4682307"
server_key_iv: "5b8aa432580f9e521e39d3f69b1b8230"
server_key_version: "encryptionkey@0"
I20260620 07:21:05.386086 1091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:05.390412 1091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:05.393563 1091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:05.393761 1091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:05.400569 1091 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:35417
I20260620 07:21:05.400583 1158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:35417 every 8 connection(s)
I20260620 07:21:05.401481 1091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:21:05.404146 1159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:05.407763 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 1091
I20260620 07:21:05.407917 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:21:05.408203 11758 external_mini_cluster.cc:1428] Setting key 968e4ba65a213f4cd59ca643ce42092d
I20260620 07:21:05.409894 1159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488: Bootstrap starting.
I20260620 07:21:05.412321 1159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488: Neither blocks nor log segments found. Creating new log.
I20260620 07:21:05.413185 1159 log.cc:824] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488: Log is configured to *not* fsync() on all Append() calls
I20260620 07:21:05.413436 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:05.409549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55803 (local address 127.11.123.190:35417)
0620 07:21:05.409978 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:05.410000 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:05.410047 (+ 47us) server_negotiation.cc:407] Connection header received
0620 07:21:05.410635 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:05.410650 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:05.410917 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:05.411179 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:05.411974 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:05.411979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:05.411990 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:21:05.411999 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:05.412047 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:21:05.412066 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:05.412192 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:05.412365 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 07:21:05.412524 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":131,"threads_started":1}
I20260620 07:21:05.415251 1159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488: No bootstrap required, opened a new log
I20260620 07:21:05.416227 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:21:05.419236 1159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52aae42f9925435c9e81d6649b336488" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35417 } }
I20260620 07:21:05.419541 1159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:21:05.419601 1159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52aae42f9925435c9e81d6649b336488, State: Initialized, Role: FOLLOWER
I20260620 07:21:05.419818 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:05.418241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34485 (local address 127.11.123.190:35417)
0620 07:21:05.418350 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:05.418354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:05.418473 (+ 119us) server_negotiation.cc:407] Connection header received
0620 07:21:05.418584 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:05.418586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:05.418611 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:05.418667 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:05.419335 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:05.419338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:05.419340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:21:05.419349 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:05.419372 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:21:05.419382 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:05.419458 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:05.419620 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 07:21:05.419719 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:21:05.420148 1159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52aae42f9925435c9e81d6649b336488" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35417 } }
I20260620 07:21:05.420290 1159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:21:05.420383 1159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:21:05.420485 1159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:05.421471 1159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52aae42f9925435c9e81d6649b336488" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35417 } }
I20260620 07:21:05.421798 1159 leader_election.cc:302] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52aae42f9925435c9e81d6649b336488; no voters:
I20260620 07:21:05.422088 1159 leader_election.cc:288] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:21:05.422276 1169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:21:05.422571 1169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [term 1 LEADER]: Becoming Leader. State: Replica: 52aae42f9925435c9e81d6649b336488, State: Running, Role: LEADER
I20260620 07:21:05.422907 1169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52aae42f9925435c9e81d6649b336488" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35417 } }
I20260620 07:21:05.423317 1159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:21:05.424467 1170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52aae42f9925435c9e81d6649b336488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52aae42f9925435c9e81d6649b336488" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35417 } } }
I20260620 07:21:05.424594 1170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:05.424994 1178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:21:05.424950 1171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52aae42f9925435c9e81d6649b336488. Latest consensus state: current_term: 1 leader_uuid: "52aae42f9925435c9e81d6649b336488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52aae42f9925435c9e81d6649b336488" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 35417 } } }
I20260620 07:21:05.425115 1171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:05.428083 1178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:21:05.433894 1178 catalog_manager.cc:1347] Generated new cluster ID: d003901a88704c6da25c0c48b5bf00ec
I20260620 07:21:05.433964 1178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:21:05.457119 1178 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:21:05.457533 1178 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:21:05.474352 1178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52aae42f9925435c9e81d6649b336488: Generated new TSK 0
I20260620 07:21:05.475191 1178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:21:05.483497 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 1091
2026-06-20T07:21:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:21:05 dist-test-slave-j663 krb5kdc[1190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:21:05 dist-test-slave-j663 krb5kdc[1190](info): set up 2 sockets
Jun 20 07:21:05 dist-test-slave-j663 krb5kdc[1190](info): commencing operation
krb5kdc: starting...
W20260620 07:21:06.561398 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:21:06 dist-test-slave-j663 krb5kdc[1190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:21:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:21:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:21:06.684423 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:44925
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:44925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:21:06.795646 1203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:06.795925 1203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:06.795969 1203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:21:06.800076 1203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:06.800140 1203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:06.800160 1203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:06.800176 1203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:06.800217 1203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:06.805557 1203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:44925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:44925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:06.806627 1203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:06.807649 1203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:06.816645 1208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:06.816737 1203 server_base.cc:1034] running on GCE node
W20260620 07:21:06.816636 1209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:06.816905 1211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:06.817595 1203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:06.818858 1203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:06.820043 1203 hybrid_clock.cc:648] HybridClock initialized: now 1781940066820017 us; error 46 us; skew 500 ppm
Jun 20 07:21:06 dist-test-slave-j663 krb5kdc[1190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940066, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:06.822757 1203 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:21:06.823558 1203 webserver.cc:460] Webserver started at http://127.11.123.190:43143/ using document root <none> and password file <none>
I20260620 07:21:06.824062 1203 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:06.824120 1203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:06.824285 1203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:06.826308 1203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "23e97f44c1574849852b90a99b17ffd6"
format_stamp: "Formatted at 2026-06-20 07:21:06 on dist-test-slave-j663"
server_key: "b9debc8c4e74ae0e7b1cef76b72a77d1"
server_key_iv: "e865062b57d7063dda109789a4d28f8e"
server_key_version: "encryptionkey@0"
I20260620 07:21:06.826809 1203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "23e97f44c1574849852b90a99b17ffd6"
format_stamp: "Formatted at 2026-06-20 07:21:06 on dist-test-slave-j663"
server_key: "b9debc8c4e74ae0e7b1cef76b72a77d1"
server_key_iv: "e865062b57d7063dda109789a4d28f8e"
server_key_version: "encryptionkey@0"
I20260620 07:21:06.830448 1203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:21:06.832760 1218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:06.834105 1203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:21:06.834228 1203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "23e97f44c1574849852b90a99b17ffd6"
format_stamp: "Formatted at 2026-06-20 07:21:06 on dist-test-slave-j663"
server_key: "b9debc8c4e74ae0e7b1cef76b72a77d1"
server_key_iv: "e865062b57d7063dda109789a4d28f8e"
server_key_version: "encryptionkey@0"
I20260620 07:21:06.834314 1203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:06.855068 1203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:06.858421 1203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:06.858577 1203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:06.865932 1203 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:44925
I20260620 07:21:06.865955 1270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:44925 every 8 connection(s)
I20260620 07:21:06.866871 1203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:21:06.869801 1271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:06.870322 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 1203
I20260620 07:21:06.870473 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:21:06.870797 11758 external_mini_cluster.cc:1428] Setting key 93f496a6645e84245136c55c9d005dfb
I20260620 07:21:06.877717 1271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6: Bootstrap starting.
I20260620 07:21:06.880338 1271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6: Neither blocks nor log segments found. Creating new log.
Jun 20 07:21:06 dist-test-slave-j663 krb5kdc[1190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:21:06.881397 1271 log.cc:824] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6: Log is configured to *not* fsync() on all Append() calls
I20260620 07:21:06.883817 1271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6: No bootstrap required, opened a new log
I20260620 07:21:06.887223 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:06.872211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55985 (local address 127.11.123.190:44925)
0620 07:21:06.872669 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:06.872689 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:06.872747 (+ 58us) server_negotiation.cc:407] Connection header received
0620 07:21:06.873586 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:06.873613 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:06.874090 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:06.874487 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:06.875141 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:06.876827 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:06.877764 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:06.878483 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:06.878662 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:06.881488 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:06.881494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:06.881512 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:06.881541 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:06.883628 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:06.884151 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:06.884157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:06.884164 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:06.884314 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:06.884674 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:06.884679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:06.884681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:06.885313 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:06.885522 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:06.885859 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 07:21:06.886181 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":119,"threads_started":1}
I20260620 07:21:06.888048 1271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e97f44c1574849852b90a99b17ffd6" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44925 } }
I20260620 07:21:06.888335 1271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:21:06.888402 1271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e97f44c1574849852b90a99b17ffd6, State: Initialized, Role: FOLLOWER
I20260620 07:21:06.888855 1271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e97f44c1574849852b90a99b17ffd6" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44925 } }
I20260620 07:21:06.889016 1271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:21:06.889104 1271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:21:06.889211 1271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:06.890103 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:21:06.890338 1271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e97f44c1574849852b90a99b17ffd6" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44925 } }
I20260620 07:21:06.890689 1271 leader_election.cc:302] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23e97f44c1574849852b90a99b17ffd6; no voters:
I20260620 07:21:06.891026 1271 leader_election.cc:288] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:21:06.891101 1276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:21:06.891312 1276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [term 1 LEADER]: Becoming Leader. State: Replica: 23e97f44c1574849852b90a99b17ffd6, State: Running, Role: LEADER
I20260620 07:21:06.891600 1276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e97f44c1574849852b90a99b17ffd6" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44925 } }
I20260620 07:21:06.892258 1271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:21:06.892737 1276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23e97f44c1574849852b90a99b17ffd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e97f44c1574849852b90a99b17ffd6" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44925 } } }
I20260620 07:21:06.892853 1276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:06.893057 1276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23e97f44c1574849852b90a99b17ffd6. Latest consensus state: current_term: 1 leader_uuid: "23e97f44c1574849852b90a99b17ffd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e97f44c1574849852b90a99b17ffd6" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 44925 } } }
I20260620 07:21:06.893246 1276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:06.893260 1284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:21:06.896503 1284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:21:06.902571 1284 catalog_manager.cc:1347] Generated new cluster ID: effaab8523304fa68f4b3a8855fe03f8
I20260620 07:21:06.902635 1284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:21:06.904006 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:06.893920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38155 (local address 127.11.123.190:44925)
0620 07:21:06.894048 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:06.894055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:06.894953 (+ 898us) server_negotiation.cc:407] Connection header received
0620 07:21:06.895078 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:06.895081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:06.895127 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:06.895208 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:06.896025 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:06.896799 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:06.898148 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:06.898507 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:06.898626 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:06.900320 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:06.900325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:06.900328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:06.900363 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:06.902139 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:06.902671 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:06.902676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:06.902680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:06.902767 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:06.903101 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:06.903104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:06.903106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:06.903342 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:06.903480 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:06.903726 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 07:21:06.903845 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:21:06.926342 1284 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:21:06.927358 1284 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:21:06.931157 1284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23e97f44c1574849852b90a99b17ffd6: Generated new TSK 0
I20260620 07:21:06.931945 1284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:21:06.943498 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 1203
2026-06-20T07:21:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:21:06 dist-test-slave-j663 krb5kdc[1302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:21:06 dist-test-slave-j663 krb5kdc[1302](info): set up 2 sockets
Jun 20 07:21:06 dist-test-slave-j663 krb5kdc[1302](info): commencing operation
krb5kdc: starting...
W20260620 07:21:07.989404 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:21:08 dist-test-slave-j663 krb5kdc[1302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:21:08.110159 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:41435
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:41435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:21:08.221117 1315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:08.221411 1315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:08.221489 1315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:21:08.225462 1315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:08.225553 1315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:08.225591 1315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:08.225651 1315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:08.225690 1315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:08.230868 1315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:41435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:41435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:08.231900 1315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:08.232916 1315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:08.241252 1320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:08.241252 1321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:08.241231 1323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:08.241297 1315 server_base.cc:1034] running on GCE node
I20260620 07:21:08.241977 1315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:08.243075 1315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:08.244232 1315 hybrid_clock.cc:648] HybridClock initialized: now 1781940068244200 us; error 45 us; skew 500 ppm
Jun 20 07:21:08 dist-test-slave-j663 krb5kdc[1302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940068, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:08.246996 1315 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:21:08.247797 1315 webserver.cc:460] Webserver started at http://127.11.123.190:39289/ using document root <none> and password file <none>
I20260620 07:21:08.248307 1315 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:08.248363 1315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:08.248528 1315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:08.250588 1315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "4ee21b1615d946eeb5b01a83a8d2f5c9"
format_stamp: "Formatted at 2026-06-20 07:21:08 on dist-test-slave-j663"
server_key: "d9d4972f7584fd91087c40b53f41f3f9"
server_key_iv: "8eed12b69c66291ddc315b4ab1defcfd"
server_key_version: "encryptionkey@0"
I20260620 07:21:08.251082 1315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "4ee21b1615d946eeb5b01a83a8d2f5c9"
format_stamp: "Formatted at 2026-06-20 07:21:08 on dist-test-slave-j663"
server_key: "d9d4972f7584fd91087c40b53f41f3f9"
server_key_iv: "8eed12b69c66291ddc315b4ab1defcfd"
server_key_version: "encryptionkey@0"
I20260620 07:21:08.254828 1315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:21:08.257334 1330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:08.258625 1315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:21:08.258771 1315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "4ee21b1615d946eeb5b01a83a8d2f5c9"
format_stamp: "Formatted at 2026-06-20 07:21:08 on dist-test-slave-j663"
server_key: "d9d4972f7584fd91087c40b53f41f3f9"
server_key_iv: "8eed12b69c66291ddc315b4ab1defcfd"
server_key_version: "encryptionkey@0"
I20260620 07:21:08.258886 1315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:08.287518 1315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:08.290864 1315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:08.291075 1315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:08.298188 1315 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:41435
I20260620 07:21:08.298213 1382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:41435 every 8 connection(s)
I20260620 07:21:08.299118 1315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:21:08.301735 1383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:08.305923 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 1315
I20260620 07:21:08.306062 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:21:08.306336 11758 external_mini_cluster.cc:1428] Setting key f3febd055faed7bb22566a9f156bd9d3
I20260620 07:21:08.307247 1383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9: Bootstrap starting.
I20260620 07:21:08.309499 1383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9: Neither blocks nor log segments found. Creating new log.
I20260620 07:21:08.310227 1383 log.cc:824] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:21:08.311960 1383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9: No bootstrap required, opened a new log
I20260620 07:21:08.314904 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:08.307689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42457 (local address 127.11.123.190:41435)
0620 07:21:08.308119 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:08.308133 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:08.308167 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:21:08.308764 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:08.308781 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:08.309180 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:08.309509 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:08.309924 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:08.310942 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:08.311840 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:08.312346 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:08.312515 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:08.313358 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:08.313364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:08.313376 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:21:08.313737 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:08.313790 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:21:08.313813 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:08.313919 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:08.314057 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 07:21:08.314251 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":128,"threads_started":1}
I20260620 07:21:08.315485 1383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee21b1615d946eeb5b01a83a8d2f5c9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 41435 } }
I20260620 07:21:08.315711 1383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:21:08.315797 1383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee21b1615d946eeb5b01a83a8d2f5c9, State: Initialized, Role: FOLLOWER
I20260620 07:21:08.316258 1383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee21b1615d946eeb5b01a83a8d2f5c9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 41435 } }
I20260620 07:21:08.316403 1383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:21:08.316473 1383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:21:08.316594 1383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:08.317534 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:21:08.317667 1383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee21b1615d946eeb5b01a83a8d2f5c9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 41435 } }
I20260620 07:21:08.317996 1383 leader_election.cc:302] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ee21b1615d946eeb5b01a83a8d2f5c9; no voters:
I20260620 07:21:08.318300 1383 leader_election.cc:288] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:21:08.318466 1388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:21:08.318897 1388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [term 1 LEADER]: Becoming Leader. State: Replica: 4ee21b1615d946eeb5b01a83a8d2f5c9, State: Running, Role: LEADER
I20260620 07:21:08.319283 1388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee21b1615d946eeb5b01a83a8d2f5c9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 41435 } }
I20260620 07:21:08.319604 1383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:21:08.320329 1388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ee21b1615d946eeb5b01a83a8d2f5c9. Latest consensus state: current_term: 1 leader_uuid: "4ee21b1615d946eeb5b01a83a8d2f5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee21b1615d946eeb5b01a83a8d2f5c9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 41435 } } }
I20260620 07:21:08.320518 1388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:08.320741 1389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ee21b1615d946eeb5b01a83a8d2f5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee21b1615d946eeb5b01a83a8d2f5c9" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 41435 } } }
I20260620 07:21:08.320976 1401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:21:08.320911 1389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:08.324535 1401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:21:08.330258 1401 catalog_manager.cc:1347] Generated new cluster ID: d0d9d4c636414ac2b31a77410f3e6c58
I20260620 07:21:08.330320 1401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:21:08.330999 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:08.323435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44651 (local address 127.11.123.190:41435)
0620 07:21:08.323565 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:08.323571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:08.326388 (+ 2817us) server_negotiation.cc:407] Connection header received
0620 07:21:08.326492 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:08.326495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:08.326545 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:08.326627 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:08.327031 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:08.327828 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:08.329005 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:08.329348 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:08.329453 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:08.330381 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:08.330385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:08.330390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:21:08.330405 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:08.330433 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:21:08.330447 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:08.330514 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:08.330817 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 07:21:08.330868 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 07:21:08.353437 1401 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:21:08.354526 1401 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:21:08.363958 1401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ee21b1615d946eeb5b01a83a8d2f5c9: Generated new TSK 0
I20260620 07:21:08.364689 1401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:21:08.371690 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 1315
2026-06-20T07:21:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:21:08 dist-test-slave-j663 krb5kdc[1414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:21:08 dist-test-slave-j663 krb5kdc[1414](info): set up 2 sockets
Jun 20 07:21:08 dist-test-slave-j663 krb5kdc[1414](info): commencing operation
krb5kdc: starting...
W20260620 07:21:09.421526 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:21:09 dist-test-slave-j663 krb5kdc[1414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:21:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:21:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:21:09.541616 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:43275
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:43275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:21:09.654827 1427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:09.655088 1427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:09.655129 1427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:21:09.659205 1427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:09.659263 1427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:09.659283 1427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:09.659299 1427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:09.659314 1427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:09.664489 1427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:43275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:43275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:09.665596 1427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:09.666572 1427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:09.675020 1433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:09.675099 1435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:09.675290 1432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:09.675115 1427 server_base.cc:1034] running on GCE node
I20260620 07:21:09.675837 1427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:09.676833 1427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:09.677991 1427 hybrid_clock.cc:648] HybridClock initialized: now 1781940069677974 us; error 44 us; skew 500 ppm
Jun 20 07:21:09 dist-test-slave-j663 krb5kdc[1414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940069, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:09.680786 1427 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:21:09.681736 1427 webserver.cc:460] Webserver started at http://127.11.123.190:44631/ using document root <none> and password file <none>
I20260620 07:21:09.682521 1427 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:09.682619 1427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:09.682868 1427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:09.684823 1427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "bae1c01d3ce04cb997f847fceb46fdbc"
format_stamp: "Formatted at 2026-06-20 07:21:09 on dist-test-slave-j663"
server_key: "ed6e8817ea126955c7fcf08a441444ac"
server_key_iv: "2cb5a2a594d0592dee7b11d1e2c7fb2d"
server_key_version: "encryptionkey@0"
I20260620 07:21:09.685392 1427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "bae1c01d3ce04cb997f847fceb46fdbc"
format_stamp: "Formatted at 2026-06-20 07:21:09 on dist-test-slave-j663"
server_key: "ed6e8817ea126955c7fcf08a441444ac"
server_key_iv: "2cb5a2a594d0592dee7b11d1e2c7fb2d"
server_key_version: "encryptionkey@0"
I20260620 07:21:09.689414 1427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:21:09.691866 1442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:09.693176 1427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:21:09.693297 1427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "bae1c01d3ce04cb997f847fceb46fdbc"
format_stamp: "Formatted at 2026-06-20 07:21:09 on dist-test-slave-j663"
server_key: "ed6e8817ea126955c7fcf08a441444ac"
server_key_iv: "2cb5a2a594d0592dee7b11d1e2c7fb2d"
server_key_version: "encryptionkey@0"
I20260620 07:21:09.693387 1427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:09.697887 1427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:09.700819 1427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:09.700959 1427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:09.707922 1427 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:43275
I20260620 07:21:09.707942 1494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:43275 every 8 connection(s)
I20260620 07:21:09.708806 1427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:21:09.711474 1495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:09.716738 1495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc: Bootstrap starting.
I20260620 07:21:09.717849 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 1427
I20260620 07:21:09.717994 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:21:09.718259 11758 external_mini_cluster.cc:1428] Setting key c744a23dc038437fedd6daa06e3e6e86
I20260620 07:21:09.719523 1495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc: Neither blocks nor log segments found. Creating new log.
I20260620 07:21:09.720252 1495 log.cc:824] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:21:09.722543 1495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc: No bootstrap required, opened a new log
I20260620 07:21:09.723804 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:09.719463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36221 (local address 127.11.123.190:43275)
0620 07:21:09.719981 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:09.719996 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:09.720032 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:21:09.720858 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:09.720885 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:09.721315 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:09.721608 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:09.722485 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:09.722490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:09.722501 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:21:09.722512 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:09.722566 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:21:09.722586 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:09.722813 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:09.722936 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 07:21:09.723097 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":177,"threads_started":1}
I20260620 07:21:09.726260 1495 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae1c01d3ce04cb997f847fceb46fdbc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43275 } }
I20260620 07:21:09.726502 1495 raft_consensus.cc:374] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:21:09.726604 1495 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae1c01d3ce04cb997f847fceb46fdbc, State: Initialized, Role: FOLLOWER
I20260620 07:21:09.726764 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:21:09.727087 1495 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae1c01d3ce04cb997f847fceb46fdbc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43275 } }
I20260620 07:21:09.727212 1495 raft_consensus.cc:388] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:21:09.727293 1495 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:21:09.727442 1495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:09.728483 1495 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae1c01d3ce04cb997f847fceb46fdbc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43275 } }
I20260620 07:21:09.728860 1495 leader_election.cc:302] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bae1c01d3ce04cb997f847fceb46fdbc; no voters:
I20260620 07:21:09.729226 1495 leader_election.cc:288] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:21:09.729328 1505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:21:09.729542 1505 raft_consensus.cc:686] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [term 1 LEADER]: Becoming Leader. State: Replica: bae1c01d3ce04cb997f847fceb46fdbc, State: Running, Role: LEADER
I20260620 07:21:09.729872 1505 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae1c01d3ce04cb997f847fceb46fdbc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43275 } }
I20260620 07:21:09.730471 1495 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:21:09.730937 1505 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae1c01d3ce04cb997f847fceb46fdbc. Latest consensus state: current_term: 1 leader_uuid: "bae1c01d3ce04cb997f847fceb46fdbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae1c01d3ce04cb997f847fceb46fdbc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43275 } } }
I20260620 07:21:09.731340 1506 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bae1c01d3ce04cb997f847fceb46fdbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae1c01d3ce04cb997f847fceb46fdbc" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 43275 } } }
I20260620 07:21:09.731416 1505 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:09.731964 1506 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:09.732379 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:09.728864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36387 (local address 127.11.123.190:43275)
0620 07:21:09.728975 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:09.728980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:09.729240 (+ 260us) server_negotiation.cc:407] Connection header received
0620 07:21:09.729414 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:09.729420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:09.729462 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:09.729532 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:09.731762 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:09.731765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:09.731768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:21:09.731780 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:09.731809 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:21:09.731822 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:09.731910 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:09.732200 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 07:21:09.732247 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 07:21:09.735296 1521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:21:09.735361 1521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:21:09.735445 1514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:21:09.736109 1514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:21:09.741012 1514 catalog_manager.cc:1347] Generated new cluster ID: 5219058faee14716a412f30dd6c1d7ae
I20260620 07:21:09.741092 1514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:21:09.746968 1514 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:21:09.747272 1514 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:21:09.762841 1514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bae1c01d3ce04cb997f847fceb46fdbc: Generated new TSK 0
I20260620 07:21:09.763655 1514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:21:09.768576 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 1427
2026-06-20T07:21:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1395 ms)
[----------] 18 tests from AuthTokenIssuingTest (25711 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:21:09 dist-test-slave-j663 krb5kdc[1526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:21:09 dist-test-slave-j663 krb5kdc[1526](info): set up 2 sockets
Jun 20 07:21:09 dist-test-slave-j663 krb5kdc[1526](info): commencing operation
krb5kdc: starting...
W20260620 07:21:10.805502 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:21:10 dist-test-slave-j663 krb5kdc[1526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:21:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:21:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:21:10.922390 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:37643
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:37643,127.11.123.189:34375,127.11.123.188:41255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:21:11.035485 1539 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:21:11.035743 1539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:11.035802 1539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:11.035882 1539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:21:11.039906 1539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:11.039958 1539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:11.039978 1539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:11.039994 1539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:11.040053 1539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:11.045063 1539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:37643,127.11.123.189:34375,127.11.123.188:41255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:37643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:11.046100 1539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:11.047101 1539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:11.055258 1545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:11.055346 1544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:11.055418 1539 server_base.cc:1034] running on GCE node
W20260620 07:21:11.055466 1547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:11.056083 1539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:11.057142 1539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:11.058388 1539 hybrid_clock.cc:648] HybridClock initialized: now 1781940071058370 us; error 30 us; skew 500 ppm
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940071, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:11.061858 1539 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:21:11.062784 1539 webserver.cc:460] Webserver started at http://127.11.123.190:43735/ using document root <none> and password file <none>
I20260620 07:21:11.063341 1539 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:11.063434 1539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:11.063660 1539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:11.065718 1539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "d4b21ef968e94d58b8acc86622f19454"
format_stamp: "Formatted at 2026-06-20 07:21:11 on dist-test-slave-j663"
server_key: "5ee46dbdb6d30dd944dc72d4e25d17b6"
server_key_iv: "4def4331dd541585d44b60b398c1f9e6"
server_key_version: "encryptionkey@0"
I20260620 07:21:11.066314 1539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "d4b21ef968e94d58b8acc86622f19454"
format_stamp: "Formatted at 2026-06-20 07:21:11 on dist-test-slave-j663"
server_key: "5ee46dbdb6d30dd944dc72d4e25d17b6"
server_key_iv: "4def4331dd541585d44b60b398c1f9e6"
server_key_version: "encryptionkey@0"
I20260620 07:21:11.069969 1539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:21:11.072294 1554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:11.073488 1539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:21:11.073622 1539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "d4b21ef968e94d58b8acc86622f19454"
format_stamp: "Formatted at 2026-06-20 07:21:11 on dist-test-slave-j663"
server_key: "5ee46dbdb6d30dd944dc72d4e25d17b6"
server_key_iv: "4def4331dd541585d44b60b398c1f9e6"
server_key_version: "encryptionkey@0"
I20260620 07:21:11.073732 1539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:11.125428 1539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:11.128295 1539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:11.128522 1539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:11.135010 1539 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:37643
I20260620 07:21:11.135283 1606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:37643 every 8 connection(s)
I20260620 07:21:11.135951 1539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:21:11.138576 1607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:11.142279 1607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:11.146205 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 1539
I20260620 07:21:11.146345 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:21:11.146620 11758 external_mini_cluster.cc:1428] Setting key 74ce47979cf927f36ef658fec8773d9c
WARNING: no policy specified for kudu/127.11.123.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.189@KRBTEST.COM" created.
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940071, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:21:11.164662 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.144528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.11.123.190:37643)
0620 07:21:11.145105 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.145126 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.145333 (+ 207us) server_negotiation.cc:407] Connection header received
0620 07:21:11.146541 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.146545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.146760 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.146907 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.148051 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.149472 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.150443 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.151122 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.151315 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.159666 (+ 8351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.159675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.159704 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.159740 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.161580 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.162204 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.162208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.162210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.162311 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.162688 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.162691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.162692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.163066 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.163167 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.163638 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.163860 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":388,"thread_start_us":137,"threads_started":1}
I20260620 07:21:11.164665 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.144528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37643 (local address 127.0.0.1:59058)
0620 07:21:11.145194 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.145230 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:11.146164 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:11.146958 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:11.146965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:11.147483 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:11.147884 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.147906 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.149609 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.149612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.150312 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.150320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.151291 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.151294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.151376 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.157725 (+ 6349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:11.157750 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:11.159481 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:11.161699 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.161710 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.161724 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.162081 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.162409 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.162415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.162417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.162581 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.163188 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:11.163197 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:11.163378 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 07:21:11.163581 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 07:21:11.163947 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":52,"spinlock_wait_cycles":55680,"thread_start_us":164,"threads_started":1}
I20260620 07:21:11.167405 1607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:11.168303 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.167953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:34375 (local address 127.0.0.1:53538)
0620 07:21:11.168126 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.168213 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 07:21:11.168522 1555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.123.189:34375: connect: Connection refused (error 111)
I20260620 07:21:11.169512 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.169294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:34375 (local address 127.0.0.1:53550)
0620 07:21:11.169413 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.169464 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:21:11.169642 1607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.189:34375: Network error: Client connection negotiation failed: client connection to 127.11.123.189:34375: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.189.keytab.
Entry for principal kudu/127.11.123.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.189.keytab.
I20260620 07:21:11.175899 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.189:34375
--webserver_interface=127.11.123.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.189.keytab
--principal=kudu/127.11.123.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:37643,127.11.123.189:34375,127.11.123.188:41255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:21:11.218912 1607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } attempt: 1
I20260620 07:21:11.220026 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.219572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:34375 (local address 127.0.0.1:53564)
0620 07:21:11.219808 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.219903 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 07:21:11.220844 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.220601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:34375 (local address 127.0.0.1:53578)
0620 07:21:11.220729 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.220782 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 07:21:11.221129 1607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.189:34375: Network error: Client connection negotiation failed: client connection to 127.11.123.189:34375: connect: Connection refused (error 111)
W20260620 07:21:11.288837 1614 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:21:11.289213 1614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:11.289288 1614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:11.289368 1614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 07:21:11.289445 1607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } attempt: 2
I20260620 07:21:11.290308 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.289976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:34375 (local address 127.0.0.1:53594)
0620 07:21:11.290148 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.290220 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 07:21:11.291148 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.290923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:34375 (local address 127.0.0.1:53600)
0620 07:21:11.291036 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.291088 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:21:11.291291 1607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.189:34375: Network error: Client connection negotiation failed: client connection to 127.11.123.189:34375: connect: Connection refused (error 111)
W20260620 07:21:11.293970 1614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:11.294023 1614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:11.294042 1614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:11.294083 1614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:11.294116 1614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:11.299563 1614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:37643,127.11.123.189:34375,127.11.123.188:41255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.189.keytab
--principal=kudu/127.11.123.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.189:34375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.123.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:11.300700 1614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:11.301797 1614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:11.310389 1622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:11.310407 1619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:11.310539 1614 server_base.cc:1034] running on GCE node
W20260620 07:21:11.310391 1620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:11.311328 1614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:11.312510 1614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:11.313721 1614 hybrid_clock.cc:648] HybridClock initialized: now 1781940071313703 us; error 37 us; skew 500 ppm
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940071, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:11.316807 1614 init.cc:377] Logged in from keytab as kudu/127.11.123.189@KRBTEST.COM (short username kudu)
I20260620 07:21:11.317837 1614 webserver.cc:460] Webserver started at http://127.11.123.189:37315/ using document root <none> and password file <none>
I20260620 07:21:11.318531 1614 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:11.318631 1614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:11.318888 1614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:11.321030 1614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/data/instance:
uuid: "8b5f4835532c4e70a3cff3e7e1a30d79"
format_stamp: "Formatted at 2026-06-20 07:21:11 on dist-test-slave-j663"
server_key: "14463471989e026bc6ce04f70ca9238e"
server_key_iv: "27c8930f38c0a3fcd555b712505a2129"
server_key_version: "encryptionkey@0"
I20260620 07:21:11.321671 1614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/wal/instance:
uuid: "8b5f4835532c4e70a3cff3e7e1a30d79"
format_stamp: "Formatted at 2026-06-20 07:21:11 on dist-test-slave-j663"
server_key: "14463471989e026bc6ce04f70ca9238e"
server_key_iv: "27c8930f38c0a3fcd555b712505a2129"
server_key_version: "encryptionkey@0"
I20260620 07:21:11.325589 1614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:21:11.328039 1629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:11.329360 1614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:21:11.329530 1614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/wal
uuid: "8b5f4835532c4e70a3cff3e7e1a30d79"
format_stamp: "Formatted at 2026-06-20 07:21:11 on dist-test-slave-j663"
server_key: "14463471989e026bc6ce04f70ca9238e"
server_key_iv: "27c8930f38c0a3fcd555b712505a2129"
server_key_version: "encryptionkey@0"
I20260620 07:21:11.329659 1614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:11.345311 1614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:11.348654 1614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:11.348865 1614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:11.356009 1614 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.189:34375
I20260620 07:21:11.356426 1681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.189:34375 every 8 connection(s)
I20260620 07:21:11.356984 1614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
I20260620 07:21:11.359658 1682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:11.363480 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 1614
I20260620 07:21:11.363338 1682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:11.363616 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-1/wal/instance
I20260620 07:21:11.363853 11758 external_mini_cluster.cc:1428] Setting key 3e6c1e5bb2b42841ece42edd268309a4
WARNING: no policy specified for kudu/127.11.123.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.188@KRBTEST.COM" created.
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940071, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.189@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:21:11.381707 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.365364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.11.123.190:37643)
0620 07:21:11.365532 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.365540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.366389 (+ 849us) server_negotiation.cc:407] Connection header received
0620 07:21:11.367379 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.367382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.367471 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.367566 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.369510 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.370047 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.371859 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.372125 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.372212 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.377564 (+ 5352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.377567 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.377569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.377596 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.379174 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.379770 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.379776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.379781 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.379887 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.380247 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.380254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.380258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.380528 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.380643 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.381285 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.381439 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:21:11.382283 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.365737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37643 (local address 127.0.0.1:59064)
0620 07:21:11.366217 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.366267 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:11.367128 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:11.367779 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:11.367791 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:11.368344 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:11.369305 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.369330 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.370194 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.370198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.371711 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.371728 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.372296 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.372301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.372535 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.373283 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:11.373310 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:11.377377 (+ 4067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:11.379313 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.379317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.379330 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.379619 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.379980 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.379987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.379989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.380139 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.380634 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:11.380639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:11.380941 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 07:21:11.381170 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 07:21:11.381436 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":151,"threads_started":1}
I20260620 07:21:11.382505 1607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } attempt: 3
I20260620 07:21:11.384280 1682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940071, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.189@KRBTEST.COM for kudu/127.11.123.189@KRBTEST.COM
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940071, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.188.keytab.
Entry for principal kudu/127.11.123.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.188.keytab.
I20260620 07:21:11.394011 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.188:41255
--webserver_interface=127.11.123.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.188.keytab
--principal=kudu/127.11.123.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:37643,127.11.123.189:34375,127.11.123.188:41255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:21:11.394048 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.384776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:34375 (local address 127.0.0.1:53620)
0620 07:21:11.384995 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.385009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:11.385103 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:11.385319 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:11.385322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:11.385488 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:11.385559 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.385564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.386276 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.386282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.386843 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.386851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.387484 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.387488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.387571 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.388048 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:11.388068 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:11.389422 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:11.392538 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.392541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.392542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.392716 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.393021 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.393023 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.393025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.393154 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.393579 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:11.393582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:11.393653 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 07:21:11.393738 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:21:11.393874 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 07:21:11.394090 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.382935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:34375 (local address 127.0.0.1:53604)
0620 07:21:11.383085 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.383100 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:11.383180 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:11.383921 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:11.383924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:11.384122 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:11.384227 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.384233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.385701 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.385706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.386715 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.386725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.387853 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.387856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.387966 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.388412 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:11.388423 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:11.389992 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:11.392231 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.392237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.392241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.392441 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.392793 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.392797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.392799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.392953 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.393547 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:11.393550 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:11.393650 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 07:21:11.393769 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 07:21:11.393913 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 07:21:11.394260 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.383226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.11.123.189:34375)
0620 07:21:11.383549 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.383563 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.383585 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:21:11.383628 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.383633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.383793 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.383941 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.384360 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.385574 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.386836 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.387405 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.387853 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.390171 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.390175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.390179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.390208 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.392096 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.392541 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.392547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.392548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.392646 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.393107 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.393113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.393117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.393427 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.393580 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.393897 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.394040 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":223,"thread_start_us":148,"threads_started":1}
I20260620 07:21:11.394716 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.384800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.11.123.189:34375)
0620 07:21:11.385046 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.385055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.385068 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:21:11.385176 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.385180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.385213 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.385310 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.385665 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.386177 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.386954 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.387375 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.387488 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.389620 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.389633 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.389703 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.389728 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.392120 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.392812 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.392817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.392819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.392897 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.393230 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.393235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.393237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.393471 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.394219 (+ 748us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.394308 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.394445 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":183,"thread_start_us":89,"threads_started":1}
I20260620 07:21:11.396178 1607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:11.396335 1682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:11.396955 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.396724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55084)
0620 07:21:11.396833 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.396908 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 07:21:11.397162 1632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
I20260620 07:21:11.397279 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.396691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55072)
0620 07:21:11.397165 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.397215 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 07:21:11.397944 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.397741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55086)
0620 07:21:11.397844 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.397889 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 07:21:11.398187 1682 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:41255: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
I20260620 07:21:11.398551 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.398220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55102)
0620 07:21:11.398408 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.398471 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260620 07:21:11.398824 1607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:41255: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
I20260620 07:21:11.430012 1607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } attempt: 1
I20260620 07:21:11.430917 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.430617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55110)
0620 07:21:11.430772 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.430841 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 07:21:11.431636 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.431430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55118)
0620 07:21:11.431538 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.431588 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 07:21:11.431768 1607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:41255: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
I20260620 07:21:11.447448 1682 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } attempt: 1
I20260620 07:21:11.448355 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.448013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55132)
0620 07:21:11.448186 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.448261 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 07:21:11.449162 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.448880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55146)
0620 07:21:11.449019 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.449096 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 07:21:11.449340 1682 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:41255: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
I20260620 07:21:11.506983 1607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } attempt: 2
I20260620 07:21:11.507827 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.507535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55158)
0620 07:21:11.507679 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.507740 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:21:11.507891 1689 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:21:11.508291 1689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:11.508360 1689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:11.508420 1689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 07:21:11.508651 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.508432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55162)
0620 07:21:11.508545 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.508599 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 07:21:11.508854 1607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:41255: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
W20260620 07:21:11.512717 1689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:11.512770 1689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:11.512789 1689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:11.512833 1689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:11.512867 1689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:11.517494 1682 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } attempt: 2
I20260620 07:21:11.518199 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.517925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55178)
0620 07:21:11.518062 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.518122 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 07:21:11.518532 1689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:37643,127.11.123.189:34375,127.11.123.188:41255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.188.keytab
--principal=kudu/127.11.123.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.188:41255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.123.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:11.519021 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.518806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55194)
0620 07:21:11.518912 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.518967 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:21:11.519155 1682 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:41255: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41255: connect: Connection refused (error 111)
I20260620 07:21:11.519732 1689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:11.520720 1689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:11.529023 1696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:11.529016 1695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:11.529258 1689 server_base.cc:1034] running on GCE node
W20260620 07:21:11.529014 1698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:11.529937 1689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:11.531034 1689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:11.532197 1689 hybrid_clock.cc:648] HybridClock initialized: now 1781940071532165 us; error 45 us; skew 500 ppm
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940071, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:11.535034 1689 init.cc:377] Logged in from keytab as kudu/127.11.123.188@KRBTEST.COM (short username kudu)
I20260620 07:21:11.535830 1689 webserver.cc:460] Webserver started at http://127.11.123.188:42471/ using document root <none> and password file <none>
I20260620 07:21:11.536378 1689 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:11.536468 1689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:11.536690 1689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:11.538676 1689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/data/instance:
uuid: "7d11b12ffd8c484aa6860f8d747e6636"
format_stamp: "Formatted at 2026-06-20 07:21:11 on dist-test-slave-j663"
server_key: "a2e4c904118d90e820153d7e60426b55"
server_key_iv: "ca05e6cfd135e5bfcbc3743b2ea5c5fe"
server_key_version: "encryptionkey@0"
I20260620 07:21:11.539227 1689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/wal/instance:
uuid: "7d11b12ffd8c484aa6860f8d747e6636"
format_stamp: "Formatted at 2026-06-20 07:21:11 on dist-test-slave-j663"
server_key: "a2e4c904118d90e820153d7e60426b55"
server_key_iv: "ca05e6cfd135e5bfcbc3743b2ea5c5fe"
server_key_version: "encryptionkey@0"
I20260620 07:21:11.542713 1689 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 07:21:11.544898 1705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:11.546087 1689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:21:11.546223 1689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/wal
uuid: "7d11b12ffd8c484aa6860f8d747e6636"
format_stamp: "Formatted at 2026-06-20 07:21:11 on dist-test-slave-j663"
server_key: "a2e4c904118d90e820153d7e60426b55"
server_key_iv: "ca05e6cfd135e5bfcbc3743b2ea5c5fe"
server_key_version: "encryptionkey@0"
I20260620 07:21:11.546329 1689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:11.581393 1689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:11.584621 1689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:11.584830 1689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:11.591835 1689 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.188:41255
I20260620 07:21:11.592121 1757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.188:41255 every 8 connection(s)
I20260620 07:21:11.592857 1689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
I20260620 07:21:11.595583 1758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:11.599205 1758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:11.601713 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 1689
I20260620 07:21:11.601848 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939794025788-11758-0/minicluster-data/master-2/wal/instance
I20260620 07:21:11.602139 11758 external_mini_cluster.cc:1428] Setting key 88cee32e3ba7bac20a3f17544a68417f
I20260620 07:21:11.608016 1607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } attempt: 3
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940071, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.188@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.188@KRBTEST.COM
I20260620 07:21:11.610317 1682 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } attempt: 3
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940071, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.188@KRBTEST.COM
I20260620 07:21:11.614198 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.600992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.11.123.190:37643)
0620 07:21:11.601157 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.601161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.602077 (+ 916us) server_negotiation.cc:407] Connection header received
0620 07:21:11.603280 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.603284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.603338 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.603442 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.604496 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.605303 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.606531 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.606773 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.606874 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.609665 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.609669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.609672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.609702 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.611734 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.612424 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.612428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.612430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.612513 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.612953 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.612957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.612958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.613208 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.613278 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.613925 (+ 647us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.614038 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:21:11.615733 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.601355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:37643 (local address 127.0.0.1:59066)
0620 07:21:11.601899 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.601957 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:11.603046 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:11.603505 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:11.603511 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:11.603965 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:11.604335 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.604354 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.605408 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.605410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.606412 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.606422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.606912 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.606915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.607138 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.607737 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:11.607761 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:11.609532 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:11.611859 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.611870 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.611880 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.612294 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.612635 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.612641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.612646 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.612841 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.613360 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:11.613366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:11.613644 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 07:21:11.614377 (+ 733us) client_negotiation.cc:240] Negotiation successful
0620 07:21:11.614714 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":83,"spinlock_wait_cycles":3840,"thread_start_us":108,"threads_started":1}
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940071, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.189@KRBTEST.COM for kudu/127.11.123.188@KRBTEST.COM
I20260620 07:21:11.616654 1762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.602796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.11.123.188:41255)
0620 07:21:11.603129 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.603137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.603169 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:21:11.603389 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.603396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.603605 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.603739 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.604594 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.605713 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.606754 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.607239 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.607318 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.610255 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.610262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.610276 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.610312 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.613116 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.613631 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.613637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.613639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.613729 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.614230 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.614234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.614235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.614525 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.614716 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.615805 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.616468 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":258,"thread_start_us":77,"threads_started":1}
I20260620 07:21:11.617537 1758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:11.619827 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.608442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55206)
0620 07:21:11.608627 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.608642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:11.608723 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:11.609143 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:11.609149 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:11.609353 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:11.609466 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.609477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.610424 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.610429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.611708 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.611717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.612319 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.612323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.612401 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.612856 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:11.612873 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:11.614359 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:11.617003 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.617006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.617008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.617292 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.618098 (+ 806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.618102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.618104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.618198 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.619341 (+ 1143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:11.619347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:11.619444 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 07:21:11.619525 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 07:21:11.619656 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 07:21:11.620378 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.610775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55218)
0620 07:21:11.611199 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.611214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:11.611297 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:11.612237 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:11.612242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:11.612444 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:11.612554 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.612563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.613638 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.613641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.614402 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.614413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.615007 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.615010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.615093 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.615488 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:11.615501 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:11.616838 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:11.618819 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.618822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.618824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.619072 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.619365 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.619368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.619371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.619466 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.619900 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:11.619903 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:11.620009 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 07:21:11.620095 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 07:21:11.620225 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330}
I20260620 07:21:11.620787 1763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.608493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55206 (local address 127.11.123.188:41255)
0620 07:21:11.608900 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.608919 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.608928 (+ 9us) server_negotiation.cc:407] Connection header received
0620 07:21:11.608953 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.608954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.608983 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.609048 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.609601 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.610316 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.611824 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.612186 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.612627 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.615035 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.615041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.615047 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.615072 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.616876 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.617882 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.617886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.617889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.617979 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.618999 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.619004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.619006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.619223 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.620264 (+ 1041us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.620332 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.620428 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"spinlock_wait_cycles":7296,"thread_start_us":116,"threads_started":1}
I20260620 07:21:11.620987 1764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.610954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.11.123.188:41255)
0620 07:21:11.611985 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.611995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.612010 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:21:11.612044 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.612046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.612083 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.612659 (+ 576us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.612744 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.613515 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.614537 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.614874 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.615293 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.616978 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.616982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.616988 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.617012 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.618713 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.619187 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.619190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.619192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.619264 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.619585 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.619588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.619590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.619780 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.620648 (+ 868us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.620718 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.620821 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":958,"thread_start_us":97,"threads_started":1}
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940071, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.188@KRBTEST.COM for kudu/127.11.123.189@KRBTEST.COM
I20260620 07:21:11.624306 1607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454: Bootstrap starting.
I20260620 07:21:11.624305 1682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79: Bootstrap starting.
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:21:11.627036 1682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79: Neither blocks nor log segments found. Creating new log.
I20260620 07:21:11.627041 1607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454: Neither blocks nor log segments found. Creating new log.
I20260620 07:21:11.627915 1682 log.cc:824] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79: Log is configured to *not* fsync() on all Append() calls
I20260620 07:21:11.627923 1607 log.cc:824] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454: Log is configured to *not* fsync() on all Append() calls
I20260620 07:21:11.628526 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.618217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.11.123.189:34375)
0620 07:21:11.618334 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.618339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.618920 (+ 581us) server_negotiation.cc:407] Connection header received
0620 07:21:11.619051 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.619054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.619099 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.619149 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.619717 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.620441 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.621507 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.621864 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.621931 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.624112 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.624119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.624128 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.624156 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.625964 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.626473 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.626478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.626480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.626564 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.626895 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.626900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.626901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.627487 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.627827 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.628224 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.628363 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:21:11.629026 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.617995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:34375 (local address 127.0.0.1:53624)
0620 07:21:11.618844 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.618854 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:11.618944 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:11.619279 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:11.619281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:11.619455 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:11.619566 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.619572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.620578 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.620583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.621388 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.621396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.622067 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.622071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.622172 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.622594 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:11.622607 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:11.623955 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:11.626090 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.626093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.626096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.626347 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.626687 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.626693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.626698 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.626784 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.627919 (+ 1135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:11.627923 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:11.628029 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 07:21:11.628714 (+ 685us) client_negotiation.cc:240] Negotiation successful
0620 07:21:11.628847 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797}
I20260620 07:21:11.629747 1758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:11.630040 1607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454: No bootstrap required, opened a new log
I20260620 07:21:11.630040 1682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79: No bootstrap required, opened a new log
I20260620 07:21:11.630833 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.619822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59080 (local address 127.11.123.190:37643)
0620 07:21:11.619920 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.619923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.620068 (+ 145us) server_negotiation.cc:407] Connection header received
0620 07:21:11.620211 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.620214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.620251 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.620294 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.621316 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.622015 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.623133 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.623491 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.623564 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.626735 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.626742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.626745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.626767 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.628466 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.628963 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.628966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.628968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.629050 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.629419 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.629424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.629426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.629640 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.629791 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.630544 (+ 753us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.630667 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 07:21:11.634789 1607 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } }
I20260620 07:21:11.635118 1607 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b21ef968e94d58b8acc86622f19454, State: Initialized, Role: FOLLOWER
I20260620 07:21:11.635061 1682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } }
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940071, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.188@KRBTEST.COM for kudu/127.11.123.188@KRBTEST.COM
I20260620 07:21:11.635262 1682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5f4835532c4e70a3cff3e7e1a30d79, State: Initialized, Role: FOLLOWER
I20260620 07:21:11.635632 1607 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } }
I20260620 07:21:11.635630 1682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } }
I20260620 07:21:11.637006 1607 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:21:11.637006 1682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 07:21:11 dist-test-slave-j663 krb5kdc[1526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.189@KRBTEST.COM
I20260620 07:21:11.639514 1763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.630233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.11.123.188:41255)
0620 07:21:11.630336 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.630341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.630550 (+ 209us) server_negotiation.cc:407] Connection header received
0620 07:21:11.630682 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.630685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.630721 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.630763 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.631241 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.631951 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.632937 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.633351 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.633440 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.635726 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.635730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.635732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.635754 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.637762 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.638233 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.638239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.638243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.638334 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.638601 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.638604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.638607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.638805 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.638881 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.639192 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.639321 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 07:21:11.639968 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.630203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41255 (local address 127.0.0.1:55230)
0620 07:21:11.630497 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:11.630511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:11.630597 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:11.630871 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:11.630874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:11.631036 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:11.631122 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.631132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.632073 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.632076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.632827 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:11.632837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.633577 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.633580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:11.633663 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.634137 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:11.634151 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:11.635602 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:11.637883 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.637887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.637889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.638135 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.638438 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:11.638440 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:11.638442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:11.638516 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:11.638945 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:11.638947 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:11.639028 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 07:21:11.639637 (+ 609us) client_negotiation.cc:240] Negotiation successful
0620 07:21:11.639796 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":112,"threads_started":1}
W20260620 07:21:11.643030 1790 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:21:11.643163 1790 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:21:11.643944 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.632299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.11.123.189:34375)
0620 07:21:11.632393 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.632397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.632558 (+ 161us) server_negotiation.cc:407] Connection header received
0620 07:21:11.632708 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.632710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.632747 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.632793 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.633742 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.634540 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.635719 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.636092 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.636180 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.639155 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.639159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.639162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.639183 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.642012 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.642505 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.642509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.642510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.642594 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.642934 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.642937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.642939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.643159 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.643327 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.643671 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.643796 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 07:21:11.644994 1758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636: Bootstrap starting.
I20260620 07:21:11.645233 1769 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } } }
I20260620 07:21:11.645443 1769 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 07:21:11.645702 1791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:21:11.645778 1791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:21:11.647789 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:21:11.649097 1581 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d4b21ef968e94d58b8acc86622f19454"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59080
I20260620 07:21:11.649310 1581 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 07:21:11.649428 1581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:11.649844 1768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } } }
I20260620 07:21:11.649981 1768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:11.650767 1581 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } }
I20260620 07:21:11.651641 1581 leader_election.cc:288] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [CANDIDATE]: Term 1 election: Requested vote from peers 8b5f4835532c4e70a3cff3e7e1a30d79 (127.11.123.189:34375), 7d11b12ffd8c484aa6860f8d747e6636 (127.11.123.188:41255)
I20260620 07:21:11.648295 1614 master_runner.cc:186] Error getting master registration for 127.11.123.188:41255: OK, instance_id { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" instance_seqno: 1781940071589620 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:21:11.652135 1614 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:21:11.652304 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4b21ef968e94d58b8acc86622f19454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79"
I20260620 07:21:11.652515 1656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:11.654014 1758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636: Neither blocks nor log segments found. Creating new log.
I20260620 07:21:11.654896 1758 log.cc:824] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636: Log is configured to *not* fsync() on all Append() calls
I20260620 07:21:11.657058 1758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636: No bootstrap required, opened a new log
I20260620 07:21:11.654332 1539 master_runner.cc:186] Error getting master registration for 127.11.123.188:41255: OK, instance_id { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" instance_seqno: 1781940071589620 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:21:11.657531 1539 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:21:11.659847 1656 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4b21ef968e94d58b8acc86622f19454 in term 1.
I20260620 07:21:11.660297 1555 leader_election.cc:302] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b5f4835532c4e70a3cff3e7e1a30d79, d4b21ef968e94d58b8acc86622f19454; no voters:
I20260620 07:21:11.660604 1769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:21:11.660895 1769 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [term 1 LEADER]: Becoming Leader. State: Replica: d4b21ef968e94d58b8acc86622f19454, State: Running, Role: LEADER
I20260620 07:21:11.661281 1769 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } }
I20260620 07:21:11.661872 1758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } }
I20260620 07:21:11.662102 1758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d11b12ffd8c484aa6860f8d747e6636, State: Initialized, Role: FOLLOWER
I20260620 07:21:11.662415 1758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } }
I20260620 07:21:11.663043 1769 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b21ef968e94d58b8acc86622f19454. Latest consensus state: current_term: 1 leader_uuid: "d4b21ef968e94d58b8acc86622f19454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } } }
I20260620 07:21:11.663139 1769 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:11.663790 1758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:21:11.663870 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4b21ef968e94d58b8acc86622f19454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7d11b12ffd8c484aa6860f8d747e6636"
I20260620 07:21:11.664058 1732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:11.665485 1732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4b21ef968e94d58b8acc86622f19454 in term 1.
W20260620 07:21:11.669154 1814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:21:11.669231 1814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:21:11.671111 1803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:21:11.671924 1803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:21:11.673866 1801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } } }
I20260620 07:21:11.673997 1801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:11.681106 1732 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [term 1 FOLLOWER]: Refusing update from remote peer d4b21ef968e94d58b8acc86622f19454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:21:11.681766 1769 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:21:11.682760 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.665835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.11.123.190:37643)
0620 07:21:11.665931 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.665936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.673368 (+ 7432us) server_negotiation.cc:407] Connection header received
0620 07:21:11.673543 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.673546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.673603 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.673661 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.674723 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.675450 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.676528 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.676887 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.676960 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.678913 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.678917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.678919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.678945 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.680668 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.681280 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.681284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.681286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.681372 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.681706 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.681709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.681711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.681937 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.682056 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.682514 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.682621 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 07:21:11.685495 1801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b21ef968e94d58b8acc86622f19454. Latest consensus state: current_term: 1 leader_uuid: "d4b21ef968e94d58b8acc86622f19454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } } }
I20260620 07:21:11.685611 1801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:11.686532 1656 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [term 1 FOLLOWER]: Refusing update from remote peer d4b21ef968e94d58b8acc86622f19454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:21:11.687449 1800 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:21:11.690716 1769 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4b21ef968e94d58b8acc86622f19454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } } }
I20260620 07:21:11.690840 1769 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:11.690959 1801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4b21ef968e94d58b8acc86622f19454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } } }
I20260620 07:21:11.691066 1801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:11.691250 1768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b21ef968e94d58b8acc86622f19454. Latest consensus state: current_term: 1 leader_uuid: "d4b21ef968e94d58b8acc86622f19454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } } }
I20260620 07:21:11.691373 1768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:11.691447 1818 mvcc.cc:205] Tried to move back new op lower bound from 7298826533584080896 to 7298826533527937024. Current Snapshot: MvccSnapshot[applied={T|T < 7298826533584080896}]
I20260620 07:21:11.692226 1819 mvcc.cc:205] Tried to move back new op lower bound from 7298826533584080896 to 7298826533527937024. Current Snapshot: MvccSnapshot[applied={T|T < 7298826533584080896}]
I20260620 07:21:11.693590 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.665447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.11.123.189:34375)
0620 07:21:11.665564 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.665568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.673331 (+ 7763us) server_negotiation.cc:407] Connection header received
0620 07:21:11.673511 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.673514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.673560 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.673610 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.685797 (+ 12187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.686573 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.687723 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.688060 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.688129 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.689800 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.689803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.689806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.689832 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.691465 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.692028 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.692032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.692034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.692112 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.692379 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.692382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.692384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.692580 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.692692 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.693330 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.693439 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:21:11.694936 1803 catalog_manager.cc:1347] Generated new cluster ID: 0c28dfe09bbb44faad415ef0e23ca0e4
I20260620 07:21:11.695003 1803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:21:11.699126 1768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4b21ef968e94d58b8acc86622f19454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } } }
I20260620 07:21:11.699229 1768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:11.699600 1800 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4b21ef968e94d58b8acc86622f19454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b21ef968e94d58b8acc86622f19454" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 37643 } } peers { permanent_uuid: "8b5f4835532c4e70a3cff3e7e1a30d79" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 34375 } } peers { permanent_uuid: "7d11b12ffd8c484aa6860f8d747e6636" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41255 } } }
I20260620 07:21:11.699780 1800 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454 [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:11.700098 1822 mvcc.cc:205] Tried to move back new op lower bound from 7298826533584080896 to 7298826533527937024. Current Snapshot: MvccSnapshot[applied={T|T < 7298826533584080896}]
I20260620 07:21:11.708482 1763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:11.666460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.11.123.188:41255)
0620 07:21:11.666706 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.666710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.676751 (+ 10041us) server_negotiation.cc:407] Connection header received
0620 07:21:11.676915 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.676921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.676966 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.677014 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.677990 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.678779 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.686241 (+ 7462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:11.686555 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:11.686637 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:11.695358 (+ 8721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:11.695362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:11.695365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:11.695391 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:11.697054 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.697542 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.697545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.697547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.697627 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:11.707318 (+ 9691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:11.707325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:11.707328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:11.707593 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:11.707735 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:11.708169 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 07:21:11.708319 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260620 07:21:11.711750 1803 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:21:11.712986 1803 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:21:11.730393 1803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4b21ef968e94d58b8acc86622f19454: Generated new TSK 0
I20260620 07:21:11.731230 1803 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:21:11.770287 1687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:11.767138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.11.123.189:34375)
0620 07:21:11.767289 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.767294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.769166 (+ 1872us) server_negotiation.cc:407] Connection header received
0620 07:21:11.769381 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.769385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.769437 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.769492 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.770144 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53654: BlockingRecv error: recv got EOF from 127.0.0.1:53654 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 07:21:11.771829 1609 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:11.767913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.11.123.190:37643)
0620 07:21:11.768026 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.768031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.770951 (+ 2920us) server_negotiation.cc:407] Connection header received
0620 07:21:11.771121 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.771124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.771173 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.771222 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.771701 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59108: BlockingRecv error: recv got EOF from 127.0.0.1:59108 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 07:21:11.773300 1763 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:11.768572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.11.123.188:41255)
0620 07:21:11.768686 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.768690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.772409 (+ 3719us) server_negotiation.cc:407] Connection header received
0620 07:21:11.772571 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.772575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.772625 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.772675 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.773185 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55256: BlockingRecv error: recv got EOF from 127.0.0.1:55256 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 07:21:11.777793 1687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:11.776151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.11.123.189:34375)
0620 07:21:11.776249 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.776254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.776758 (+ 504us) server_negotiation.cc:407] Connection header received
0620 07:21:11.776993 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.776997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.777072 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.777155 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.777690 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53662: BlockingRecv error: recv got EOF from 127.0.0.1:53662 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 07:21:11.782794 1687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:11.781413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.11.123.189:34375)
0620 07:21:11.781508 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.781513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.781966 (+ 453us) server_negotiation.cc:407] Connection header received
0620 07:21:11.782137 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.782140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.782188 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.782237 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.782698 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53674: BlockingRecv error: recv got EOF from 127.0.0.1:53674 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 07:21:11.788780 1687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:11.787068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.11.123.189:34375)
0620 07:21:11.787170 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.787175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.787761 (+ 586us) server_negotiation.cc:407] Connection header received
0620 07:21:11.788019 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.788022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.788074 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.788126 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.788650 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53682: BlockingRecv error: recv got EOF from 127.0.0.1:53682 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 07:21:11.797842 1687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:11.796102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.11.123.189:34375)
0620 07:21:11.796266 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.796271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.796802 (+ 531us) server_negotiation.cc:407] Connection header received
0620 07:21:11.797011 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.797014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.797097 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.797155 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.797730 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53694: BlockingRecv error: recv got EOF from 127.0.0.1:53694 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 07:21:11.810029 1687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:11.808411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53710 (local address 127.11.123.189:34375)
0620 07:21:11.808531 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.808536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.809015 (+ 479us) server_negotiation.cc:407] Connection header received
0620 07:21:11.809275 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.809278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.809331 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.809385 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.809913 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53710: BlockingRecv error: recv got EOF from 127.0.0.1:53710 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 07:21:11.830680 1687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:11.828960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.11.123.189:34375)
0620 07:21:11.829113 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.829117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.829650 (+ 533us) server_negotiation.cc:407] Connection header received
0620 07:21:11.829862 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.829866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.829920 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.829975 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.830566 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53724: BlockingRecv error: recv got EOF from 127.0.0.1:53724 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 07:21:11.867328 1687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:11.865656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.11.123.189:34375)
0620 07:21:11.865793 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.865798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.866339 (+ 541us) server_negotiation.cc:407] Connection header received
0620 07:21:11.866541 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.866545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.866598 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.866650 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.867208 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53732: BlockingRecv error: recv got EOF from 127.0.0.1:53732 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 07:21:11.935318 1687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:11.934201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.11.123.189:34375)
0620 07:21:11.934350 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:11.934355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:11.934541 (+ 186us) server_negotiation.cc:407] Connection header received
0620 07:21:11.934643 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:11.934645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:11.934692 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:11.934770 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:11.935195 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53744: BlockingRecv error: recv got EOF from 127.0.0.1:53744 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 07:21:12.066670 1687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:12.065419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.11.123.189:34375)
0620 07:21:12.065557 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:12.065561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:12.065796 (+ 235us) server_negotiation.cc:407] Connection header received
0620 07:21:12.065942 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:12.065946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:12.066006 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:12.066097 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:12.066552 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53748: BlockingRecv error: recv got EOF from 127.0.0.1:53748 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 07:21:12.325953 1687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:12.324834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.11.123.189:34375)
0620 07:21:12.325031 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:12.325036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:12.325050 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:21:12.325235 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:12.325238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:12.325296 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:12.325388 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:12.325841 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53760: BlockingRecv error: recv got EOF from 127.0.0.1:53760 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 07:21:12.647040 1791 catalog_manager.cc:1259] Loaded cluster ID: 0c28dfe09bbb44faad415ef0e23ca0e4
I20260620 07:21:12.647127 1791 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79: loading cluster ID for follower catalog manager: success
I20260620 07:21:12.648792 1791 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79: acquiring CA information for follower catalog manager: success
I20260620 07:21:12.649983 1791 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b5f4835532c4e70a3cff3e7e1a30d79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:21:12.670588 1814 catalog_manager.cc:1259] Loaded cluster ID: 0c28dfe09bbb44faad415ef0e23ca0e4
I20260620 07:21:12.670688 1814 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636: loading cluster ID for follower catalog manager: success
I20260620 07:21:12.672523 1814 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636: acquiring CA information for follower catalog manager: success
I20260620 07:21:12.673746 1814 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d11b12ffd8c484aa6860f8d747e6636: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:21:12.844033 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:12.840255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.11.123.189:34375)
0620 07:21:12.840491 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:12.840495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:12.840575 (+ 80us) server_negotiation.cc:407] Connection header received
0620 07:21:12.840701 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:12.840704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:12.840873 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:12.840991 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:21:12.841280 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:12.841824 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:12.842649 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:12.842903 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:12.842984 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:12.843263 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:21:12.843691 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:21:12.843782 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:12.843862 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 07:21:12.843910 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20260620 07:21:12.851017 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:12.847099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.11.123.188:41255)
0620 07:21:12.847362 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:12.847366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:12.847378 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:21:12.847457 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:12.847459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:12.847574 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:12.847669 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:21:12.847941 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:12.848497 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:12.849385 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:12.849695 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:12.849780 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:12.850075 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:21:12.850615 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:21:12.850736 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:12.850764 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 07:21:12.850818 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":112,"threads_started":1}
I20260620 07:21:12.858070 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:12.854119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.11.123.189:34375)
0620 07:21:12.854247 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:12.854252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:12.854492 (+ 240us) server_negotiation.cc:407] Connection header received
0620 07:21:12.854692 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:12.854697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:12.854755 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:12.854804 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:21:12.855182 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:12.855943 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:12.856884 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:12.857190 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:12.857285 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:12.857556 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:21:12.857631 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:21:12.857710 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:12.857856 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 07:21:12.857911 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:21:12.858503 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:12.854506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.11.123.188:41255)
0620 07:21:12.854607 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:12.854612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:12.854894 (+ 282us) server_negotiation.cc:407] Connection header received
0620 07:21:12.855009 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:12.855012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:12.855068 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:12.855144 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:21:12.855526 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:12.856247 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:12.857390 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:12.857696 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:12.857805 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:12.858024 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:21:12.858093 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:21:12.858160 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:12.858313 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 07:21:12.858366 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:21:12.858734 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:12.854154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.11.123.190:37643)
0620 07:21:12.854391 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:12.854396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:12.854672 (+ 276us) server_negotiation.cc:407] Connection header received
0620 07:21:12.854826 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:12.854830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:12.855006 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:12.855126 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:21:12.855359 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:12.855936 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:12.857067 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:12.857433 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:12.857523 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:12.857919 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:21:12.858354 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:21:12.858449 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:12.858531 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 07:21:12.858594 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20260620 07:21:12.861603 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 1539
I20260620 07:21:12.867944 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 1614
I20260620 07:21:12.874049 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 1689
2026-06-20T07:21:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3107 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:21:12 dist-test-slave-j663 krb5kdc[1910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:21:12 dist-test-slave-j663 krb5kdc[1910](info): set up 2 sockets
Jun 20 07:21:12 dist-test-slave-j663 krb5kdc[1910](info): commencing operation
krb5kdc: starting...
W20260620 07:21:13.917529 11758 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/test-user.keytab.
Jun 20 07:21:13 dist-test-slave-j663 krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:21:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:21:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.123.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
Entry for principal kudu/127.11.123.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab.
I20260620 07:21:14.032219 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:33065
--webserver_interface=127.11.123.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:33065,127.11.123.189:38535,127.11.123.188:41931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:21:14.139928 1923 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:21:14.140221 1923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:14.140290 1923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:14.140389 1923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:21:14.144366 1923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:14.144444 1923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:14.144482 1923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:14.144531 1923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:14.144570 1923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:14.149703 1923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:33065,127.11.123.189:38535,127.11.123.188:41931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.190.keytab
--principal=kudu/127.11.123.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.190:33065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.123.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:14.150668 1923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:14.151615 1923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:14.160043 1929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:14.160097 1928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:14.160212 1923 server_base.cc:1034] running on GCE node
W20260620 07:21:14.160048 1931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:14.160914 1923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:14.162061 1923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:14.163246 1923 hybrid_clock.cc:648] HybridClock initialized: now 1781940074163214 us; error 46 us; skew 500 ppm
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940074, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:14.165944 1923 init.cc:377] Logged in from keytab as kudu/127.11.123.190@KRBTEST.COM (short username kudu)
I20260620 07:21:14.167008 1923 webserver.cc:460] Webserver started at http://127.11.123.190:41127/ using document root <none> and password file <none>
I20260620 07:21:14.167680 1923 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:14.167768 1923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:14.167979 1923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:14.170030 1923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/data/instance:
uuid: "0b4f2b26ab174cf08094b264870da70d"
format_stamp: "Formatted at 2026-06-20 07:21:14 on dist-test-slave-j663"
server_key: "ec8a1a056bb5fae000cf954de81d42ca"
server_key_iv: "4fc5ad18eb63bd070ee7e3bd2eed51f1"
server_key_version: "encryptionkey@0"
I20260620 07:21:14.170591 1923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/wal/instance:
uuid: "0b4f2b26ab174cf08094b264870da70d"
format_stamp: "Formatted at 2026-06-20 07:21:14 on dist-test-slave-j663"
server_key: "ec8a1a056bb5fae000cf954de81d42ca"
server_key_iv: "4fc5ad18eb63bd070ee7e3bd2eed51f1"
server_key_version: "encryptionkey@0"
I20260620 07:21:14.174196 1923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:21:14.176709 1938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:14.177987 1923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:21:14.178126 1923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/wal
uuid: "0b4f2b26ab174cf08094b264870da70d"
format_stamp: "Formatted at 2026-06-20 07:21:14 on dist-test-slave-j663"
server_key: "ec8a1a056bb5fae000cf954de81d42ca"
server_key_iv: "4fc5ad18eb63bd070ee7e3bd2eed51f1"
server_key_version: "encryptionkey@0"
I20260620 07:21:14.178234 1923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:14.209604 1923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:14.212445 1923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:14.212648 1923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:14.219499 1923 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.190:33065
I20260620 07:21:14.219817 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.190:33065 every 8 connection(s)
I20260620 07:21:14.220474 1923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/data/info.pb
I20260620 07:21:14.223228 1991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:14.226907 1991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:14.227983 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 1923
I20260620 07:21:14.228113 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-0/wal/instance
I20260620 07:21:14.228400 11758 external_mini_cluster.cc:1428] Setting key c6a0302f419fd0ca2ae5bf67c23768e0
WARNING: no policy specified for kudu/127.11.123.189@KRBTEST.COM; defaulting to no policy
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940074, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.189@KRBTEST.COM" created.
I20260620 07:21:14.243091 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.229224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33065 (local address 127.0.0.1:35282)
0620 07:21:14.229787 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.229849 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:14.230702 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:14.231361 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:14.231366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:14.231806 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:14.232185 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.232205 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.233552 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.233556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.234889 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.234904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.235585 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.235589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.235719 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.236320 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:14.236344 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:14.238124 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:14.240212 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.240222 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.240232 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.240503 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.240816 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.240819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.240821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.240934 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.241584 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:14.241590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:14.241745 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 07:21:14.241959 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 07:21:14.242379 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":85,"thread_start_us":143,"threads_started":1}
I20260620 07:21:14.243090 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.229224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.11.123.190:33065)
0620 07:21:14.229785 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.229803 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.230063 (+ 260us) server_negotiation.cc:407] Connection header received
0620 07:21:14.231001 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.231005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.231190 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.231365 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.232370 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.233422 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.235031 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.235417 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.235540 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.238269 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.238274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.238284 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.238305 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.240060 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.240606 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.240612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.240614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.240719 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.241028 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.241031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.241033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.241462 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.241599 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.242053 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.242275 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":431,"spinlock_wait_cycles":56704,"thread_start_us":159,"threads_started":1}
I20260620 07:21:14.245672 1991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:14.246459 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.246160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:38535 (local address 127.0.0.1:51788)
0620 07:21:14.246294 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.246375 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:38535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:21:14.246673 1941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.123.189:38535: connect: Connection refused (error 111)
I20260620 07:21:14.247565 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.247318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:38535 (local address 127.0.0.1:51794)
0620 07:21:14.247440 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.247490 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:38535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:21:14.247730 1991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.189:38535: Network error: Client connection negotiation failed: client connection to 127.11.123.189:38535: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.189.keytab.
Entry for principal kudu/127.11.123.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.189.keytab.
I20260620 07:21:14.255640 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.189:38535
--webserver_interface=127.11.123.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.189.keytab
--principal=kudu/127.11.123.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:33065,127.11.123.189:38535,127.11.123.188:41931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:21:14.296991 1991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } attempt: 1
I20260620 07:21:14.297858 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.297548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:38535 (local address 127.0.0.1:51796)
0620 07:21:14.297707 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.297775 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:38535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 07:21:14.298655 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.298442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:38535 (local address 127.0.0.1:51812)
0620 07:21:14.298545 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.298595 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:38535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 07:21:14.298774 1991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.189:38535: Network error: Client connection negotiation failed: client connection to 127.11.123.189:38535: connect: Connection refused (error 111)
W20260620 07:21:14.362852 1998 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:21:14.363104 1998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:14.363128 1998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:14.363168 1998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 07:21:14.366920 1991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } attempt: 2
W20260620 07:21:14.367200 1998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:14.367242 1998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:14.367261 1998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:14.367306 1998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:14.367334 1998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:14.367781 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.367464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:38535 (local address 127.0.0.1:51818)
0620 07:21:14.367619 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.367689 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:38535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 07:21:14.368423 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.368208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:38535 (local address 127.0.0.1:51824)
0620 07:21:14.368311 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.368361 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.189:38535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 07:21:14.368556 1991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.189:38535: Network error: Client connection negotiation failed: client connection to 127.11.123.189:38535: connect: Connection refused (error 111)
I20260620 07:21:14.372829 1998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:33065,127.11.123.189:38535,127.11.123.188:41931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.189.keytab
--principal=kudu/127.11.123.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.189:38535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.123.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:14.373824 1998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:14.374785 1998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:14.382902 2003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:14.382937 2006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:14.382946 2004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:14.383229 1998 server_base.cc:1034] running on GCE node
I20260620 07:21:14.383733 1998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:14.384816 1998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:14.385977 1998 hybrid_clock.cc:648] HybridClock initialized: now 1781940074385937 us; error 52 us; skew 500 ppm
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940074, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:14.388595 1998 init.cc:377] Logged in from keytab as kudu/127.11.123.189@KRBTEST.COM (short username kudu)
I20260620 07:21:14.389485 1998 webserver.cc:460] Webserver started at http://127.11.123.189:43309/ using document root <none> and password file <none>
I20260620 07:21:14.390031 1998 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:14.390118 1998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:14.390344 1998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:14.392333 1998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/data/instance:
uuid: "9b1bb02cb7374f05816548d0d34ee981"
format_stamp: "Formatted at 2026-06-20 07:21:14 on dist-test-slave-j663"
server_key: "c409454cd35ba797a09184a6c8db6dd8"
server_key_iv: "69e6edaa947ef3e5420061826139630d"
server_key_version: "encryptionkey@0"
I20260620 07:21:14.392879 1998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/wal/instance:
uuid: "9b1bb02cb7374f05816548d0d34ee981"
format_stamp: "Formatted at 2026-06-20 07:21:14 on dist-test-slave-j663"
server_key: "c409454cd35ba797a09184a6c8db6dd8"
server_key_iv: "69e6edaa947ef3e5420061826139630d"
server_key_version: "encryptionkey@0"
I20260620 07:21:14.396473 1998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:21:14.398849 2013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:14.399971 1998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 07:21:14.400103 1998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/wal
uuid: "9b1bb02cb7374f05816548d0d34ee981"
format_stamp: "Formatted at 2026-06-20 07:21:14 on dist-test-slave-j663"
server_key: "c409454cd35ba797a09184a6c8db6dd8"
server_key_iv: "69e6edaa947ef3e5420061826139630d"
server_key_version: "encryptionkey@0"
I20260620 07:21:14.400207 1998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:14.412837 1998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:14.415443 1998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:14.415634 1998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:14.422317 1998 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.189:38535
I20260620 07:21:14.422576 2065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.189:38535 every 8 connection(s)
I20260620 07:21:14.423265 1998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/data/info.pb
I20260620 07:21:14.425805 2066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:14.429203 2066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:14.430831 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 1998
I20260620 07:21:14.430976 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-1/wal/instance
I20260620 07:21:14.431241 11758 external_mini_cluster.cc:1428] Setting key ee236f66f9718dbd8abbae8ce2f147f2
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940074, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.189@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.123.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.123.188@KRBTEST.COM" created.
I20260620 07:21:14.442905 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.430847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.11.123.190:33065)
0620 07:21:14.431012 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.431018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.431918 (+ 900us) server_negotiation.cc:407] Connection header received
0620 07:21:14.433267 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.433272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.433340 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.433443 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.434503 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.435377 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.436463 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.436909 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.437058 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.439273 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.439279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.439284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.439316 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.440722 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.441288 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.441292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.441293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.441363 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.441761 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.441768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.441773 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.441982 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.442060 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.442697 (+ 637us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.442789 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:21:14.443768 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.431224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33065 (local address 127.0.0.1:35298)
0620 07:21:14.431740 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.431793 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:14.433007 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:14.433583 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:14.433590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:14.433933 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:14.434350 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.434361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.435498 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.435501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.436328 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.436336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.437064 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.437066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.437245 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.437794 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:14.437810 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:14.439106 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:14.440823 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.440827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.440837 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.441203 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.441489 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.441495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.441499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.441659 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.442089 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:14.442098 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:14.442419 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 07:21:14.442641 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 07:21:14.442974 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":119,"threads_started":1}
I20260620 07:21:14.445365 2066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940074, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.189@KRBTEST.COM for kudu/127.11.123.189@KRBTEST.COM
I20260620 07:21:14.454766 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.445875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:38535 (local address 127.0.0.1:51838)
0620 07:21:14.446010 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.446020 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:14.446077 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:14.446561 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:14.446564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:14.446731 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:14.446797 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.446802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.447716 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.447719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.448215 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.448221 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.448790 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.448794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.448937 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.449344 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:14.449355 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:14.451597 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:14.453460 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.453463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.453464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.453637 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.453932 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.453934 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.453935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.454005 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.454425 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:14.454427 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:14.454499 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 07:21:14.454579 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:21:14.454665 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":37}
I20260620 07:21:14.455024 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.445974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.11.123.189:38535)
0620 07:21:14.446262 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.446270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.446286 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:21:14.446315 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.446319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.446449 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.446583 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.446900 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.447596 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.448323 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.448675 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.448785 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.451712 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.451720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.451729 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.451750 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.453343 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.453729 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.453733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.453734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.453831 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.454078 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.454080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.454081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.454322 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.454422 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.454636 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.454743 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":204,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.123.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.188.keytab.
Entry for principal kudu/127.11.123.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.188.keytab.
I20260620 07:21:14.456698 2066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:14.456784 11758 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
/tmp/dist-test-taskbzqqUD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.188:41931
--webserver_interface=127.11.123.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.188.keytab
--principal=kudu/127.11.123.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.123.148:37557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.123.190:33065,127.11.123.189:38535,127.11.123.188:41931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:21:14.457320 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.457124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52106)
0620 07:21:14.457205 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.457275 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 07:21:14.457499 2017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
I20260620 07:21:14.458230 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.458045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52108)
0620 07:21:14.458139 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.458181 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 07:21:14.458431 2066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:41931: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
I20260620 07:21:14.459695 1991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } attempt: 3
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940074, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.189@KRBTEST.COM
I20260620 07:21:14.469353 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.460111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51848 (local address 127.11.123.189:38535)
0620 07:21:14.460464 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.460470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.460486 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:21:14.460523 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.460528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.460579 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.460666 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.461025 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.461916 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.463170 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.464232 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.464358 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.466527 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.466529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.466531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.466551 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.467867 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.468327 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.468329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.468330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.468406 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.468668 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.468670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.468671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.468810 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.468878 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.469180 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.469263 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
I20260620 07:21:14.469403 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.460053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:38535 (local address 127.0.0.1:51848)
0620 07:21:14.460194 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.460209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:14.460300 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:14.460692 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:14.460695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:14.460824 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:14.460916 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.460921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.462061 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.462064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.463044 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.463054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.464346 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.464348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.464433 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.464892 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:14.464902 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:14.466401 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:14.467993 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.467996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.467998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.468240 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.468489 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.468491 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.468494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.468582 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.468889 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:14.468891 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:14.468992 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 07:21:14.469121 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 07:21:14.469249 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 07:21:14.470314 1991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:14.470907 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.470683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52110)
0620 07:21:14.470792 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.470845 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 07:21:14.471593 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.471353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52120)
0620 07:21:14.471494 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.471541 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 07:21:14.471720 1991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:41931: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
I20260620 07:21:14.502875 1991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } attempt: 1
I20260620 07:21:14.503784 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.503446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52132)
0620 07:21:14.503622 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.503694 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 07:21:14.504534 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.504287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52138)
0620 07:21:14.504412 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.504465 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 07:21:14.504693 1991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:41931: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
I20260620 07:21:14.507642 2066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } attempt: 1
I20260620 07:21:14.508302 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.508076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52144)
0620 07:21:14.508191 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.508240 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 07:21:14.508960 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.508777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52152)
0620 07:21:14.508863 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.508906 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 07:21:14.509161 2066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:41931: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
W20260620 07:21:14.568713 2073 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:21:14.568969 2073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:21:14.569036 2073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:21:14.569144 2073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:21:14.573616 2073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:21:14.573673 2073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:21:14.573716 2073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:21:14.573755 2073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:21:14.573804 2073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:21:14.577322 2066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } attempt: 2
I20260620 07:21:14.578166 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.577893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52166)
0620 07:21:14.578034 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.578093 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 07:21:14.578851 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.578674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52168)
0620 07:21:14.578757 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.578798 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 07:21:14.579077 2066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:41931: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
I20260620 07:21:14.579704 2073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.123.148:37557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.123.190:33065,127.11.123.189:38535,127.11.123.188:41931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/krb5kdc/kudu_127.11.123.188.keytab
--principal=kudu/127.11.123.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.123.188:41931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.123.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:21:14.579877 1991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } attempt: 2
I20260620 07:21:14.580515 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.580289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52172)
0620 07:21:14.580410 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.580457 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 07:21:14.581022 2073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:21:14.581233 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.580933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52176)
0620 07:21:14.581100 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.581164 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 07:21:14.581390 1991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.123.188:41931: Network error: Client connection negotiation failed: client connection to 127.11.123.188:41931: connect: Connection refused (error 111)
I20260620 07:21:14.582258 2073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:21:14.590366 2079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:14.590415 2073 server_base.cc:1034] running on GCE node
W20260620 07:21:14.590351 2078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:21:14.590353 2081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:21:14.591172 2073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:21:14.592249 2073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:21:14.593405 2073 hybrid_clock.cc:648] HybridClock initialized: now 1781940074593389 us; error 24 us; skew 500 ppm
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940074, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:21:14.596063 2073 init.cc:377] Logged in from keytab as kudu/127.11.123.188@KRBTEST.COM (short username kudu)
I20260620 07:21:14.596915 2073 webserver.cc:460] Webserver started at http://127.11.123.188:41133/ using document root <none> and password file <none>
I20260620 07:21:14.597705 2073 fs_manager.cc:359] Metadata directory not provided
I20260620 07:21:14.597776 2073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:21:14.597950 2073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:21:14.600100 2073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/data/instance:
uuid: "dd14bf59a7b34755b7f5beed280a164a"
format_stamp: "Formatted at 2026-06-20 07:21:14 on dist-test-slave-j663"
server_key: "4184b27ff61cc2ee2189022eece56470"
server_key_iv: "09ce9ba3c331aeec78896246bac8ee5c"
server_key_version: "encryptionkey@0"
I20260620 07:21:14.600610 2073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/wal/instance:
uuid: "dd14bf59a7b34755b7f5beed280a164a"
format_stamp: "Formatted at 2026-06-20 07:21:14 on dist-test-slave-j663"
server_key: "4184b27ff61cc2ee2189022eece56470"
server_key_iv: "09ce9ba3c331aeec78896246bac8ee5c"
server_key_version: "encryptionkey@0"
I20260620 07:21:14.604375 2073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:21:14.606714 2088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:21:14.607858 2073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:21:14.607977 2073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/data,/tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/wal
uuid: "dd14bf59a7b34755b7f5beed280a164a"
format_stamp: "Formatted at 2026-06-20 07:21:14 on dist-test-slave-j663"
server_key: "4184b27ff61cc2ee2189022eece56470"
server_key_iv: "09ce9ba3c331aeec78896246bac8ee5c"
server_key_version: "encryptionkey@0"
I20260620 07:21:14.608076 2073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:21:14.634032 2073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:21:14.637447 2073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:21:14.637660 2073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:21:14.645149 2073 rpc_server.cc:307] RPC server started. Bound to: 127.11.123.188:41931
I20260620 07:21:14.645471 2140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.123.188:41931 every 8 connection(s)
I20260620 07:21:14.646193 2073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/data/info.pb
I20260620 07:21:14.649204 2141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:21:14.652920 11758 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu as pid 2073
I20260620 07:21:14.653059 11758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbzqqUD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939794025788-11758-0/minicluster-data/master-2/wal/instance
I20260620 07:21:14.653451 11758 external_mini_cluster.cc:1428] Setting key 6bae9855dc36e8c40ba32804c6cf4e5a
I20260620 07:21:14.653426 2141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940074, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.188@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.188@KRBTEST.COM
I20260620 07:21:14.667467 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.655501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.11.123.190:33065)
0620 07:21:14.655648 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.655653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.656078 (+ 425us) server_negotiation.cc:407] Connection header received
0620 07:21:14.656441 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.656444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.656497 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.656618 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.657746 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.658631 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.660048 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.660410 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.660519 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.663493 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.663497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.663500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.663530 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.665412 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.665926 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.665931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.665933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.666028 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.666375 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.666381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.666383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.666598 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.666713 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.667138 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.667276 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:21:14.668422 2144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.654914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.11.123.188:41931)
0620 07:21:14.655401 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.655421 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.655465 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:21:14.656243 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.656262 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.656524 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.656709 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.657644 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.658991 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.660162 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.660728 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.660821 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.663966 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.663972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.663987 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.664007 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.665645 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.666090 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.666100 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.666101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.666199 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.666478 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.666483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.666485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.666913 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.667114 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.667298 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.667510 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":312,"spinlock_wait_cycles":3200,"thread_start_us":161,"threads_started":1}
I20260620 07:21:14.668428 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.655413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.190:33065 (local address 127.0.0.1:35308)
0620 07:21:14.655922 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.655977 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:14.656256 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:14.656681 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:14.656687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:14.657217 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:14.657590 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.657611 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.658776 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.658779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.659917 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.659928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.660578 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.660581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.660711 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.661508 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:14.661532 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:14.663337 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:14.665532 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.665544 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.665554 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.665812 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.666130 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.666134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.666136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.666262 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.666724 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:14.666734 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:14.666903 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 07:21:14.667090 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 07:21:14.667451 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":7,"spinlock_wait_cycles":3840,"thread_start_us":131,"threads_started":1}
I20260620 07:21:14.669723 2141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:14.670233 2066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } attempt: 3
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940074, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.189@KRBTEST.COM for kudu/127.11.123.188@KRBTEST.COM
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940074, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.188@KRBTEST.COM for kudu/127.11.123.189@KRBTEST.COM
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.190@KRBTEST.COM
I20260620 07:21:14.680450 2144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.670813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.11.123.188:41931)
0620 07:21:14.670930 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.670933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.671269 (+ 336us) server_negotiation.cc:407] Connection header received
0620 07:21:14.671359 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.671362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.671397 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.671481 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.671898 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.672602 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.673396 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.673730 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.673834 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.676181 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.676186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.676189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.676216 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.678701 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.679176 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.679179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.679181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.679279 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.679576 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.679580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.679581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.679765 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.679866 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.680161 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.680275 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":46}
I20260620 07:21:14.680460 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.670729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52186)
0620 07:21:14.671197 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.671207 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:14.671288 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:14.671494 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:14.671497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:14.671690 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:14.671772 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.671778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.672694 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.672700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.673289 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.673297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.673858 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.673863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.674023 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.674682 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:14.674702 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:14.676046 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:14.678811 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.678815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.678817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.679059 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.679365 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.679369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.679375 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.679475 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.679861 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:14.679867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:14.679975 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 07:21:14.680092 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 07:21:14.680239 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":31}
I20260620 07:21:14.680555 1991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } attempt: 3
I20260620 07:21:14.680516 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.670512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.11.123.189:38535)
0620 07:21:14.670905 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.670909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.670921 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:21:14.670956 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.670961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.671006 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.671081 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.671656 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.672442 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.673548 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.673872 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.673972 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.676738 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.676743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.676746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.676783 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.678547 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.679050 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.679054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.679056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.679170 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.679469 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.679473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.679475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.679780 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.679866 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.680214 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.680332 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":330}
I20260620 07:21:14.681288 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.670260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.189:38535 (local address 127.0.0.1:51858)
0620 07:21:14.670362 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.670380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:14.670458 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:14.671118 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:14.671121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:14.671313 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:14.671470 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.671480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.672532 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.672535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.673446 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.673457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.674259 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.674262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.674331 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.674859 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:14.674876 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:14.676590 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:14.678694 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.678697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.678699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.678939 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.679281 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.679285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.679287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.679388 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.679906 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:14.679909 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:14.680006 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 07:21:14.680934 (+ 928us) client_negotiation.cc:240] Negotiation successful
0620 07:21:14.681068 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 07:21:14.681929 2141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:21:14.682428 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.670712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.11.123.190:33065)
0620 07:21:14.670812 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.670816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.670829 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:21:14.670865 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.670868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.670909 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.670991 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.671823 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.672349 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.673586 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.673863 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.673972 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.677603 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.677608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.677612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.677643 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.679322 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.679802 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.679806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.679808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.679902 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.680214 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.680220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.680222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.680456 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.681235 (+ 779us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.682140 (+ 905us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.682280 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:21:14.684496 2066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981: Bootstrap starting.
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940074, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.190@KRBTEST.COM for kudu/127.11.123.188@KRBTEST.COM
I20260620 07:21:14.687604 2066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981: Neither blocks nor log segments found. Creating new log.
I20260620 07:21:14.688635 2066 log.cc:824] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940074, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.123.188@KRBTEST.COM for kudu/127.11.123.188@KRBTEST.COM
I20260620 07:21:14.690742 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.680964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52190)
0620 07:21:14.681238 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.681254 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:14.681341 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:14.681844 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:14.681847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:14.682042 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:14.682185 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.682197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.683677 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.683681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.684462 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.684470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.685034 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.685039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.685149 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.685586 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:14.685602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:14.687050 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:14.688970 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.688974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.688976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.689263 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.689585 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.689587 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.689589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.689705 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.690216 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:14.690219 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:14.690331 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 07:21:14.690442 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 07:21:14.690578 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260620 07:21:14.690749 2144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.681496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.11.123.188:41931)
0620 07:21:14.681604 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.681608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.681619 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:21:14.681652 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.681655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.681694 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.682642 (+ 948us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.682727 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.683507 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.684562 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.684906 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.685342 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.687203 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.687206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.687209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.687237 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.688852 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.689389 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.689392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.689393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.689470 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.689853 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.689860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.689862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.690104 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.690224 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.690510 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.690614 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 07:21:14.691356 2066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981: No bootstrap required, opened a new log
Jun 20 07:21:14 dist-test-slave-j663 krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781940073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.123.189@KRBTEST.COM
I20260620 07:21:14.693563 1991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d: Bootstrap starting.
I20260620 07:21:14.693706 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.682355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.123.188:41931 (local address 127.0.0.1:52194)
0620 07:21:14.682564 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 07:21:14.682576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:21:14.682640 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:21:14.684077 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:21:14.684080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:21:14.684233 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:21:14.684310 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.684316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.685480 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.685484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.686296 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:21:14.686307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.686981 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.686985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:21:14.687068 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.687507 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:21:14.687529 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:21:14.689129 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:21:14.691979 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.691984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.691987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.692190 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.692580 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:21:14.692583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:21:14.692585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:21:14.692670 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:21:14.693155 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:21:14.693161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:21:14.693250 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 07:21:14.693395 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 07:21:14.693521 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 07:21:14.694151 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.682421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.11.123.188:41931)
0620 07:21:14.683846 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.683856 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.683868 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:21:14.683900 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.683903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.683942 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.684029 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.684566 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.685357 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.686429 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.686782 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.686884 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.689253 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.689257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.689260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.689291 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.691827 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.692302 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.692305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.692307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.692389 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.692776 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.692779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.692782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.692984 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.693112 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.693927 (+ 815us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.694030 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1362,"thread_start_us":124,"threads_started":1}
I20260620 07:21:14.696354 1991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d: Neither blocks nor log segments found. Creating new log.
I20260620 07:21:14.696816 2141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a: Bootstrap starting.
I20260620 07:21:14.697255 1991 log.cc:824] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:21:14.697896 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.684630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.11.123.189:38535)
0620 07:21:14.684772 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.684776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.684926 (+ 150us) server_negotiation.cc:407] Connection header received
0620 07:21:14.685116 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.685119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.685159 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.685209 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.686365 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.687092 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.689405 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.689992 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.690306 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.693739 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.693742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.693745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.693768 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.695961 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.696439 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.696442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.696444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.696526 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.696837 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.696842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.696845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.697044 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.697282 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.697626 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.697736 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:21:14.699275 2141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a: Neither blocks nor log segments found. Creating new log.
I20260620 07:21:14.699282 1991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d: No bootstrap required, opened a new log
I20260620 07:21:14.700132 2141 log.cc:824] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:21:14.701285 11758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:21:14.701478 2066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } }
I20260620 07:21:14.701673 2066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b1bb02cb7374f05816548d0d34ee981, State: Initialized, Role: FOLLOWER
I20260620 07:21:14.701970 2066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } }
I20260620 07:21:14.702436 2150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } } }
I20260620 07:21:14.702432 1965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0b4f2b26ab174cf08094b264870da70d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35314
I20260620 07:21:14.702639 2150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:14.703326 2066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:21:14.703415 1991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } }
I20260620 07:21:14.703619 1991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4f2b26ab174cf08094b264870da70d, State: Initialized, Role: FOLLOWER
I20260620 07:21:14.703928 1991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } }
I20260620 07:21:14.704416 2153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } } }
I20260620 07:21:14.704627 2153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:14.705204 1991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:21:14.705289 1965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 07:21:14.705390 1965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:14.706756 1965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } }
I20260620 07:21:14.707749 1965 leader_election.cc:288] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [CANDIDATE]: Term 1 election: Requested vote from peers 9b1bb02cb7374f05816548d0d34ee981 (127.11.123.189:38535), dd14bf59a7b34755b7f5beed280a164a (127.11.123.188:41931)
I20260620 07:21:14.709159 2141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a: No bootstrap required, opened a new log
W20260620 07:21:14.711544 2172 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:21:14.711618 2172 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:21:14.710318 2040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b4f2b26ab174cf08094b264870da70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9b1bb02cb7374f05816548d0d34ee981"
I20260620 07:21:14.713838 2040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:14.713718 2141 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } }
I20260620 07:21:14.713956 2141 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd14bf59a7b34755b7f5beed280a164a, State: Initialized, Role: FOLLOWER
W20260620 07:21:14.714157 1944 tablet.cc:2369] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:21:14.714272 2141 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } }
I20260620 07:21:14.715175 2040 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4f2b26ab174cf08094b264870da70d in term 1.
I20260620 07:21:14.715427 2141 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:21:14.715495 2115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b4f2b26ab174cf08094b264870da70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dd14bf59a7b34755b7f5beed280a164a"
I20260620 07:21:14.715667 2115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:21:14.717049 2115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4f2b26ab174cf08094b264870da70d in term 1.
I20260620 07:21:14.717597 1942 leader_election.cc:302] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4f2b26ab174cf08094b264870da70d, dd14bf59a7b34755b7f5beed280a164a; no voters:
I20260620 07:21:14.717916 2153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:21:14.718144 2153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [term 1 LEADER]: Becoming Leader. State: Replica: 0b4f2b26ab174cf08094b264870da70d, State: Running, Role: LEADER
I20260620 07:21:14.718458 2153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } }
I20260620 07:21:14.720577 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b4f2b26ab174cf08094b264870da70d. Latest consensus state: current_term: 1 leader_uuid: "0b4f2b26ab174cf08094b264870da70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } } }
I20260620 07:21:14.720721 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:14.721138 2197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:21:14.721230 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } } }
I20260620 07:21:14.721359 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:14.722210 2197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:21:14.724821 1998 master_runner.cc:263] No leader master found from master 9b1bb02cb7374f05816548d0d34ee981
I20260620 07:21:14.724884 1998 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:21:14.709753 1923 master_runner.cc:186] Error getting master registration for 127.11.123.189:38535: OK, instance_id { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" instance_seqno: 1781940074420456 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:21:14.725785 1923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 07:21:14.720515 2193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:21:14.725947 2193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 07:21:14.731205 2178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:21:14.731333 2178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:21:14.734431 2115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [term 1 FOLLOWER]: Refusing update from remote peer 0b4f2b26ab174cf08094b264870da70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:21:14.735103 2194 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:21:14.739741 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.713275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.11.123.188:41931)
0620 07:21:14.713386 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.713391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.725354 (+ 11963us) server_negotiation.cc:407] Connection header received
0620 07:21:14.725556 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.725559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.725606 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.725660 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.726743 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.727617 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.728770 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.729192 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.729277 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.731129 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.731133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.731136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.731160 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.733024 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.735987 (+ 2963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.735991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.735993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.736090 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.737527 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.737532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.737535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.737787 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.737896 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.739431 (+ 1535us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.739571 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 07:21:14.742179 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.713366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.11.123.190:33065)
0620 07:21:14.713475 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.713479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.732455 (+ 18976us) server_negotiation.cc:407] Connection header received
0620 07:21:14.732616 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.732619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.732666 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.732731 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.733602 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.734375 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.735818 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.736164 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.736236 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.737991 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.737995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.737998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.738026 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.739987 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.740738 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.740742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.740744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.740830 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.741194 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.741197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.741199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.741421 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.741535 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.741884 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.742011 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 07:21:14.743100 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b4f2b26ab174cf08094b264870da70d. Latest consensus state: current_term: 1 leader_uuid: "0b4f2b26ab174cf08094b264870da70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } } }
I20260620 07:21:14.743235 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:14.744357 2040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [term 1 FOLLOWER]: Refusing update from remote peer 0b4f2b26ab174cf08094b264870da70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:21:14.744997 2194 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:21:14.749466 2150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b4f2b26ab174cf08094b264870da70d. Latest consensus state: current_term: 1 leader_uuid: "0b4f2b26ab174cf08094b264870da70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } } }
I20260620 07:21:14.749593 2150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:14.750294 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:14.711267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.11.123.189:38535)
0620 07:21:14.711388 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.711392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.731680 (+ 20288us) server_negotiation.cc:407] Connection header received
0620 07:21:14.732070 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.732074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.732127 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.732205 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.733897 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.734587 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.736584 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:14.736927 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:14.737010 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:14.741063 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:21:14.741067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:21:14.741070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:21:14.741121 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:21:14.742821 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.745194 (+ 2373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.745198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.745200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.745296 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:21:14.745613 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:21:14.745617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:21:14.745618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:21:14.745851 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:21:14.745982 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:14.750013 (+ 4031us) server_negotiation.cc:299] Negotiation successful
0620 07:21:14.750142 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:21:14.753105 2153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b4f2b26ab174cf08094b264870da70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } } }
I20260620 07:21:14.753242 2153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:14.753381 2198 mvcc.cc:205] Tried to move back new op lower bound from 7298826546103304192 to 7298826546049888256. Current Snapshot: MvccSnapshot[applied={T|T < 7298826546103304192}]
I20260620 07:21:14.753757 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b4f2b26ab174cf08094b264870da70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } } }
I20260620 07:21:14.753890 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:14.754348 2200 mvcc.cc:205] Tried to move back new op lower bound from 7298826546103304192 to 7298826546049888256. Current Snapshot: MvccSnapshot[applied={T|T < 7298826546103304192}]
I20260620 07:21:14.759692 2153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b4f2b26ab174cf08094b264870da70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } } }
I20260620 07:21:14.759835 2153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d [sys.catalog]: This master's current role is: LEADER
I20260620 07:21:14.760059 2197 catalog_manager.cc:1347] Generated new cluster ID: 9014a3ad477a4643969898ee9db8ac09
I20260620 07:21:14.760123 2197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:21:14.761065 2202 mvcc.cc:205] Tried to move back new op lower bound from 7298826546103304192 to 7298826546049888256. Current Snapshot: MvccSnapshot[applied={T|T < 7298826546103304192}]
I20260620 07:21:14.761252 2150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b4f2b26ab174cf08094b264870da70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4f2b26ab174cf08094b264870da70d" member_type: VOTER last_known_addr { host: "127.11.123.190" port: 33065 } } peers { permanent_uuid: "9b1bb02cb7374f05816548d0d34ee981" member_type: VOTER last_known_addr { host: "127.11.123.189" port: 38535 } } peers { permanent_uuid: "dd14bf59a7b34755b7f5beed280a164a" member_type: VOTER last_known_addr { host: "127.11.123.188" port: 41931 } } }
I20260620 07:21:14.761340 2150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:21:14.793973 2197 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:21:14.796025 2197 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:21:14.810106 2197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b4f2b26ab174cf08094b264870da70d: Generated new TSK 0
I20260620 07:21:14.811007 2197 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:21:14.848052 1994 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:14.844724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.11.123.190:33065)
0620 07:21:14.844869 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.844873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.846944 (+ 2071us) server_negotiation.cc:407] Connection header received
0620 07:21:14.847143 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.847147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.847198 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.847251 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.847914 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35342: BlockingRecv error: recv got EOF from 127.0.0.1:35342 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 07:21:14.849463 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:14.845728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.11.123.189:38535)
0620 07:21:14.845848 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.845853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.848538 (+ 2685us) server_negotiation.cc:407] Connection header received
0620 07:21:14.848702 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.848706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.848753 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.848803 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.849352 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51884: BlockingRecv error: recv got EOF from 127.0.0.1:51884 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 07:21:14.850759 2146 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:14.846152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.11.123.188:41931)
0620 07:21:14.846265 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.846270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.849840 (+ 3570us) server_negotiation.cc:407] Connection header received
0620 07:21:14.850082 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.850085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.850132 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.850182 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.850645 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52204: BlockingRecv error: recv got EOF from 127.0.0.1:52204 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 07:21:14.855132 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:14.853648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.11.123.189:38535)
0620 07:21:14.853746 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.853750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.854195 (+ 445us) server_negotiation.cc:407] Connection header received
0620 07:21:14.854411 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.854414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.854483 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.854534 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.855042 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51890: BlockingRecv error: recv got EOF from 127.0.0.1:51890 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 07:21:14.859963 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:14.858559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.11.123.189:38535)
0620 07:21:14.858652 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.858656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.859131 (+ 475us) server_negotiation.cc:407] Connection header received
0620 07:21:14.859314 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.859317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.859364 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.859412 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.859875 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51892: BlockingRecv error: recv got EOF from 127.0.0.1:51892 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 07:21:14.865393 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:14.864010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.11.123.189:38535)
0620 07:21:14.864100 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.864104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.864578 (+ 474us) server_negotiation.cc:407] Connection header received
0620 07:21:14.864738 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.864741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.864785 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.864832 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.865311 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51902: BlockingRecv error: recv got EOF from 127.0.0.1:51902 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 07:21:14.872956 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:14.871451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51908 (local address 127.11.123.189:38535)
0620 07:21:14.871573 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.871577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.872093 (+ 516us) server_negotiation.cc:407] Connection header received
0620 07:21:14.872287 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.872290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.872339 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.872390 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.872867 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51908: BlockingRecv error: recv got EOF from 127.0.0.1:51908 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 07:21:14.885334 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:14.883584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.11.123.189:38535)
0620 07:21:14.883696 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.883700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.884287 (+ 587us) server_negotiation.cc:407] Connection header received
0620 07:21:14.884499 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.884503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.884572 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.884634 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.885213 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51910: BlockingRecv error: recv got EOF from 127.0.0.1:51910 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 07:21:14.906235 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:14.904044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.11.123.189:38535)
0620 07:21:14.904173 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.904178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.904998 (+ 820us) server_negotiation.cc:407] Connection header received
0620 07:21:14.905264 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.905268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.905346 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.905411 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.906109 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51920: BlockingRecv error: recv got EOF from 127.0.0.1:51920 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 07:21:14.941799 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:14.940633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.11.123.189:38535)
0620 07:21:14.940778 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:14.940782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:14.940871 (+ 89us) server_negotiation.cc:407] Connection header received
0620 07:21:14.940976 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:14.940979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:14.941043 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:14.941171 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:14.941647 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51926: BlockingRecv error: recv got EOF from 127.0.0.1:51926 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 07:21:15.009037 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:15.007794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.11.123.189:38535)
0620 07:21:15.007965 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:15.007970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:15.008121 (+ 151us) server_negotiation.cc:407] Connection header received
0620 07:21:15.008302 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:15.008305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:15.008360 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:15.008453 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:15.008917 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51936: BlockingRecv error: recv got EOF from 127.0.0.1:51936 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 07:21:15.140298 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:15.139143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.11.123.189:38535)
0620 07:21:15.139287 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:15.139291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:15.139467 (+ 176us) server_negotiation.cc:407] Connection header received
0620 07:21:15.139555 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:15.139558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:15.139612 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:15.139728 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:15.140180 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51944: BlockingRecv error: recv got EOF from 127.0.0.1:51944 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 07:21:15.399474 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:21:15.398386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.11.123.189:38535)
0620 07:21:15.398561 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:15.398566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:15.398582 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:21:15.398733 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:15.398737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:15.398798 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:15.398894 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:21:15.399352 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51952: BlockingRecv error: recv got EOF from 127.0.0.1:51952 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:21:15.727244 2193 catalog_manager.cc:1259] Loaded cluster ID: 9014a3ad477a4643969898ee9db8ac09
I20260620 07:21:15.727334 2193 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a: loading cluster ID for follower catalog manager: success
I20260620 07:21:15.728960 2193 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a: acquiring CA information for follower catalog manager: success
I20260620 07:21:15.730077 2193 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dd14bf59a7b34755b7f5beed280a164a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:21:15.732383 2178 catalog_manager.cc:1259] Loaded cluster ID: 9014a3ad477a4643969898ee9db8ac09
I20260620 07:21:15.732436 2178 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981: loading cluster ID for follower catalog manager: success
I20260620 07:21:15.733840 2178 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981: acquiring CA information for follower catalog manager: success
I20260620 07:21:15.734778 2178 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b1bb02cb7374f05816548d0d34ee981: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:21:15.917387 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:15.913533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.11.123.189:38535)
0620 07:21:15.913796 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:15.913799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:15.913847 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:21:15.914011 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:15.914014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:15.914138 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:15.914237 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:21:15.914456 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:15.914976 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:15.915812 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:15.916042 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:15.916121 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:15.916422 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:21:15.916938 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:21:15.917073 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:15.917170 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 07:21:15.917226 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260620 07:21:15.923880 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:15.920264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.11.123.188:41931)
0620 07:21:15.920500 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:15.920504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:15.920520 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:21:15.920637 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:15.920640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:15.920755 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:15.920875 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:21:15.921123 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:15.921637 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:15.922455 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:15.922700 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:15.922807 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:15.923112 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:21:15.923488 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:21:15.923613 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:15.923674 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 07:21:15.923728 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":62,"threads_started":1}
I20260620 07:21:15.930454 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:15.926706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.11.123.188:41931)
0620 07:21:15.926855 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:15.926859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:15.927145 (+ 286us) server_negotiation.cc:407] Connection header received
0620 07:21:15.927334 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:15.927340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:15.927410 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:15.927488 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:21:15.927762 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:15.928304 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:15.929352 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:15.929634 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:15.929743 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:15.930065 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:21:15.930144 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:21:15.930211 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:15.930287 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 07:21:15.930332 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:21:15.930689 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:15.926645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.11.123.189:38535)
0620 07:21:15.926756 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:15.926760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:15.927008 (+ 248us) server_negotiation.cc:407] Connection header received
0620 07:21:15.927203 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:15.927207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:15.927265 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:15.927316 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:21:15.927692 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:15.928448 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:15.929484 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:15.929817 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:15.929919 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:15.930181 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:21:15.930266 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:21:15.930349 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:15.930505 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 07:21:15.930570 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:21:15.931313 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:21:15.926612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.11.123.190:33065)
0620 07:21:15.927141 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 07:21:15.927147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:21:15.927235 (+ 88us) server_negotiation.cc:407] Connection header received
0620 07:21:15.927431 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:21:15.927435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:21:15.927595 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:21:15.927718 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:21:15.927944 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:15.928691 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:15.929756 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:21:15.930079 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:21:15.930183 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:21:15.930613 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:21:15.930927 (+ 314us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:21:15.930989 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 07:21:15.931099 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 07:21:15.931146 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":66,"threads_started":1}
I20260620 07:21:15.934386 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 1923
I20260620 07:21:15.940739 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 1998
I20260620 07:21:15.946432 11758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbzqqUD/build/debug/bin/kudu with pid 2073
2026-06-20T07:21:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3071 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6179 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (281914 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST